aymane-maghouti / Real-Time-Streaming-Kafka-Debezium-Spark-StreamingLinks
This project demonstrates real-time data streaming and processing architecture using Kafka, Spark Streaming, and Debezium for capturing CDC (Change Data Capture) events. The pipeline collects transaction data, processes it in real time, and updates a dashboard to display real-time analytics for smartphone data.
☆11Updated last year
Alternatives and similar repositories for Real-Time-Streaming-Kafka-Debezium-Spark-Streaming
Users that are interested in Real-Time-Streaming-Kafka-Debezium-Spark-Streaming are comparing it to the libraries listed below
Sorting:
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆42Updated last year
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆36Updated last year
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆44Updated last year
- This project shows how to capture changes from postgres database and stream them into kafka☆38Updated last year
- This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data wareh…☆160Updated 2 years ago
- An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Ka…☆285Updated 8 months ago
- ☆54Updated 11 months ago
- ☆70Updated last week
- This repository contains the necessary configuration files and DAGs (Directed Acyclic Graphs) for setting up a robust data engineering en…☆22Updated last year
- This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python…☆46Updated last year
- ☆83Updated 10 months ago
- Data Engineering with AWS, 2nd edition - Published by Packt☆160Updated 2 years ago
- This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark cluste…☆11Updated 2 years ago
- ☆206Updated 2 years ago
- Sample repo for startdataengineering DE 101 free course☆69Updated last year
- With everything I learned from DEZoomcamp from datatalks.club, this project performs a batch processing on AWS for the cycling dataset wh…☆14Updated 3 years ago
- ☆114Updated last year
- YouTube tutorial project☆105Updated 2 years ago
- ☆29Updated last year
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆45Updated last year
- An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Az…☆27Updated 2 years ago
- In this project, we will build and ETL(Extract,Transform,Load) pipeline using the Spotify API on AWS. The pipeline will retrieve data fro…☆24Updated 2 years ago
- Code for blog at https://www.startdataengineering.com/post/python-for-de/☆88Updated last year
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆103Updated 7 months ago
- Are you building or do you support an e-commerce website? If so, then this content is for you! Worldwide digital sales in 2020 eclipsed …☆93Updated 3 months ago
- ☆93Updated 9 months ago
- Resources for the free AWS Data Engineering course on youtube☆102Updated 4 years ago
- ☆44Updated 10 months ago
- Realtime Data Engineering Project☆30Updated 9 months ago
- Databricks DLT Apparel Pipeline Project: Learn medallion architecture, streaming, and data engineering with Delta Live Tables. Includes s…☆26Updated 2 weeks ago