aymane-maghouti / Real-Time-Streaming-Kafka-Debezium-Spark-StreamingLinks
This project demonstrates real-time data streaming and processing architecture using Kafka, Spark Streaming, and Debezium for capturing CDC (Change Data Capture) events. The pipeline collects transaction data, processes it in real time, and updates a dashboard to display real-time analytics for smartphone data.
☆13Updated last year
Alternatives and similar repositories for Real-Time-Streaming-Kafka-Debezium-Spark-Streaming
Users that are interested in Real-Time-Streaming-Kafka-Debezium-Spark-Streaming are comparing it to the libraries listed below
Sorting:
- This project shows how to capture changes from postgres database and stream them into kafka☆39Updated last year
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆45Updated 2 years ago
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆37Updated 2 years ago
- ☆44Updated 5 years ago
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆44Updated 2 years ago
- This repository contains the necessary configuration files and DAGs (Directed Acyclic Graphs) for setting up a robust data engineering en…☆23Updated last year
- ☆70Updated last week
- An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Ka…☆303Updated 10 months ago
- This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark cluste…☆11Updated 2 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆107Updated 9 months ago
- End to end data engineering project☆57Updated 3 years ago
- ☆21Updated last year
- data-warehouse-snowflake-for-data-engineering☆18Updated 2 years ago
- This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python…☆46Updated last year
- Resources and projects from Udacity Data Engineering with AWS nano degree programme☆28Updated 2 years ago
- Resources for the free AWS Data Engineering course on youtube☆104Updated 4 years ago
- Sample repo for startdataengineering DE 101 free course☆72Updated last year
- Code for blog at https://www.startdataengineering.com/post/python-for-de/☆92Updated last year
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆13Updated 2 years ago
- Realtime Data Engineering Project☆30Updated 11 months ago
- This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data wareh…☆183Updated 2 years ago
- Simple stream processing pipeline☆110Updated last year
- Repository related to Spark SQL and Pyspark using Python3☆42Updated 3 years ago
- An AWS Data Engineering End-to-End Project (Glue, Lambda, Kinesis, Redshift, QuickSight, Athena, EC2, S3)☆16Updated 2 years ago
- With everything I learned from DEZoomcamp from datatalks.club, this project performs a batch processing on AWS for the cycling dataset wh…☆14Updated last week
- 😈Complete End to End ETL Pipeline with Spark, Airflow, & AWS☆50Updated 6 years ago
- ☆18Updated 2 years ago
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆46Updated 2 years ago
- GitHub repository related to the course Mastering Elastic Map Reduce for Data Engineers☆23Updated 3 years ago
- This repository contains an end-to-end data engineering project using Apache Flink, focused on performing sales analytics. The project de…☆11Updated 2 years ago