aymane-maghouti / Real-Time-Streaming-Kafka-Debezium-Spark-StreamingLinks
This project demonstrates real-time data streaming and processing architecture using Kafka, Spark Streaming, and Debezium for capturing CDC (Change Data Capture) events. The pipeline collects transaction data, processes it in real time, and updates a dashboard to display real-time analytics for smartphone data.
☆11Updated 10 months ago
Alternatives and similar repositories for Real-Time-Streaming-Kafka-Debezium-Spark-Streaming
Users that are interested in Real-Time-Streaming-Kafka-Debezium-Spark-Streaming are comparing it to the libraries listed below
Sorting:
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆38Updated last year
- ☆39Updated 9 months ago
- This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data wareh…☆151Updated last year
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆102Updated 5 months ago
- ☆70Updated last week
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆34Updated last year
- An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Ka…☆272Updated 6 months ago
- This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark cluste…☆11Updated last year
- This project shows how to capture changes from postgres database and stream them into kafka☆38Updated last year
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆41Updated last year
- YouTube tutorial project☆105Updated last year
- Data Engineering with AWS, 2nd edition - Published by Packt☆150Updated last year
- This repository contains a collection of SQL scripts demonstrating various analytical techniques, such as changes over time, cumulative, …☆101Updated 5 months ago
- ☆21Updated last year
- Sample repo for startdataengineering DE 101 free course☆69Updated last year
- An end-to-end data engineering pipeline that fetches real-time YouTube analytics and streams them through Kafka for processing with ksqlD…☆12Updated last year
- Resources and projects from Udacity Data Engineering with AWS nano degree programme☆28Updated 2 years ago
- In this project, we will build and ETL(Extract,Transform,Load) pipeline using the Spotify API on AWS. The pipeline will retrieve data fro…☆23Updated 2 years ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆42Updated last year
- Resources for the free AWS Data Engineering course on youtube☆100Updated 4 years ago
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆142Updated 2 years ago
- ☆204Updated 2 years ago
- An ETL pipeline that extracts weather and air quality data from public APIs, transforms the data into a clean, analyzable format, and loa…☆20Updated 11 months ago
- Nyc_Taxi_Data_Pipeline - DE Project☆120Updated 10 months ago
- ☆98Updated last year
- Data Engineering YouTube Analysis Project by Darshil Parmar☆203Updated last year
- Projects done in the Data Engineer Nanodegree Program by Udacity.com☆162Updated 2 years ago
- An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Az…☆23Updated last year
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆44Updated last year
- Realtime Data Engineering Project☆30Updated 7 months ago