aymane-maghouti / Real-Time-Streaming-Kafka-Debezium-Spark-StreamingLinks
This project demonstrates real-time data streaming and processing architecture using Kafka, Spark Streaming, and Debezium for capturing CDC (Change Data Capture) events. The pipeline collects transaction data, processes it in real time, and updates a dashboard to display real-time analytics for smartphone data.
☆11Updated last year
Alternatives and similar repositories for Real-Time-Streaming-Kafka-Debezium-Spark-Streaming
Users that are interested in Real-Time-Streaming-Kafka-Debezium-Spark-Streaming are comparing it to the libraries listed below
Sorting:
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆42Updated last year
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆36Updated last year
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆43Updated last year
- This project shows how to capture changes from postgres database and stream them into kafka☆38Updated last year
- This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark cluste…☆11Updated 2 years ago
- This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data wareh…☆167Updated 2 years ago
- An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Ka…☆287Updated 9 months ago
- ☆56Updated last year
- Nyc_Taxi_Data_Pipeline - DE Project☆129Updated last year
- Resources and projects from Udacity Data Engineering with AWS nano degree programme☆28Updated 2 years ago
- With everything I learned from DEZoomcamp from datatalks.club, this project performs a batch processing on AWS for the cycling dataset wh…☆14Updated 3 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆105Updated 7 months ago
- ☆209Updated 2 years ago
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆46Updated last year
- This project contain build end-to-end e-commerce data from data source into data warehouse and visualization.☆13Updated last year
- This is a repository to demonstrate my details, skills, projects and to keep track of my progression in Data Analytics and Data Science t…☆136Updated 10 months ago
- ☆70Updated last week
- Data Engineering with AWS, 2nd edition - Published by Packt☆164Updated 2 years ago
- Data Engineering Project with Hadoop HDFS and Kafka☆118Updated 2 years ago
- Realtime Data Engineering Project☆30Updated 10 months ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆42Updated 2 years ago
- ☆45Updated last year
- An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Az…☆27Updated 2 years ago
- This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python…☆46Updated last year
- Are you building or do you support an e-commerce website? If so, then this content is for you! Worldwide digital sales in 2020 eclipsed …☆93Updated 4 months ago
- YouTube tutorial project☆105Updated 2 years ago
- Projects done in the Data Engineer Nanodegree Program by Udacity.com☆164Updated 2 years ago
- ☆39Updated last year
- Sample repo for startdataengineering DE 101 free course☆71Updated last year
- This repository contains the necessary configuration files and DAGs (Directed Acyclic Graphs) for setting up a robust data engineering en…☆22Updated last year