aymane-maghouti / Real-Time-Streaming-Kafka-Debezium-Spark-StreamingLinks
This project demonstrates real-time data streaming and processing architecture using Kafka, Spark Streaming, and Debezium for capturing CDC (Change Data Capture) events. The pipeline collects transaction data, processes it in real time, and updates a dashboard to display real-time analytics for smartphone data.
☆11Updated 10 months ago
Alternatives and similar repositories for Real-Time-Streaming-Kafka-Debezium-Spark-Streaming
Users that are interested in Real-Time-Streaming-Kafka-Debezium-Spark-Streaming are comparing it to the libraries listed below
Sorting:
- An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Ka…☆273Updated 7 months ago
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆41Updated last year
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆35Updated last year
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆40Updated last year
- This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data wareh…☆155Updated last year
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆102Updated 5 months ago
- ☆44Updated 10 months ago
- This repo is for the Linkedin Learning course: End-to-End Data Engineering Project☆24Updated last year
- ☆203Updated 2 years ago
- This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark cluste…☆11Updated last year
- This project shows how to capture changes from postgres database and stream them into kafka☆38Updated last year
- ☆68Updated last month
- YouTube tutorial project☆106Updated last year
- This project contain build end-to-end e-commerce data from data source into data warehouse and visualization.☆13Updated last year
- Resources and projects from Udacity Data Engineering with AWS nano degree programme☆28Updated 2 years ago
- With everything I learned from DEZoomcamp from datatalks.club, this project performs a batch processing on AWS for the cycling dataset wh…☆14Updated 3 years ago
- ☆40Updated last year
- Sample repo for startdataengineering DE 101 free course☆69Updated last year
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆44Updated last year
- ☆21Updated last year
- This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python…☆44Updated last year
- In this project, we will build and ETL(Extract,Transform,Load) pipeline using the Spotify API on AWS. The pipeline will retrieve data fro…☆24Updated 2 years ago
- Realtime Data Engineering Project☆29Updated 8 months ago
- ☆289Updated last year
- Nyc_Taxi_Data_Pipeline - DE Project☆121Updated 10 months ago
- Apartments Data Pipeline using Airflow and Spark.☆22Updated 3 years ago
- ☆84Updated 7 months ago
- Data Engineering with AWS, 2nd edition - Published by Packt☆153Updated last year
- ☆24Updated last year
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆12Updated 2 years ago