An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
☆362Feb 14, 2025Updated last year
Alternatives and similar repositories for e2e-data-engineering
Users that are interested in e2e-data-engineering are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆50Dec 11, 2023Updated 2 years ago
- This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark cluste…☆11Oct 11, 2023Updated 3 years ago
- This project shows how to capture changes from postgres database and stream them into kafka☆43May 17, 2024Updated 2 years ago
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆47Jan 4, 2024Updated 2 years ago
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆40Dec 18, 2023Updated 2 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- This repository contains the necessary configuration files and DAGs (Directed Acyclic Graphs) for setting up a robust data engineering en…☆25Jan 26, 2024Updated 2 years ago
- This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data wareh…☆229Oct 23, 2023Updated 2 years ago
- An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Az…☆30Oct 2, 2023Updated 3 years ago
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆53Dec 4, 2023Updated 2 years ago
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆145Jul 27, 2023Updated 3 years ago
- Demo of using Airflow☆11Jun 24, 2022Updated 4 years ago
- PySpark Tutorial for Beginners on Google Colab: Hands-On Guide☆17Sep 13, 2020Updated 6 years ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆25May 14, 2022Updated 4 years ago
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆13Aug 26, 2023Updated 3 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python…☆47Mar 14, 2024Updated 2 years ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆43Sep 26, 2023Updated 3 years ago
- ☆16Oct 8, 2025Updated last year
- This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow or…☆24Aug 21, 2025Updated last year
- used Airflow, Postgres, Kafka, Spark, and Cassandra, and GitHub Actions to establish an end-to-end data pipeline☆32Oct 25, 2023Updated 2 years ago
- ☆35Nov 25, 2023Updated 2 years ago
- Series follows learning from Apache Spark (PySpark) with quick tips and workaround for daily problems in hand☆56Sep 30, 2023Updated 3 years ago
- An AWS Data Engineering End-to-End Project (Glue, Lambda, Kinesis, Redshift, QuickSight, Athena, EC2, S3)☆17Sep 20, 2023Updated 3 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆120Jan 8, 2026Updated 9 months ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Practical Data Engineering: A Hands-On Real-Estate Project Guide☆827Jun 25, 2026Updated 3 months ago
- 🥭 Designed and optimized a CNN architecture to accurately detect and classify 7 types of mango leaf diseases, reaching 99.21% test accur…☆14Nov 14, 2023Updated 2 years ago
- YouTube tutorial project☆113Oct 17, 2023Updated 2 years ago
- A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Doc…☆25Nov 19, 2024Updated last year
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆354Jan 12, 2022Updated 4 years ago
- StreamSoft enables real-time analysis of any stock market☆14Apr 24, 2024Updated 2 years ago
- Example end to end data engineering project.☆1,439Dec 8, 2022Updated 3 years ago
- ☆21Aug 8, 2024Updated 2 years ago
- ☆219Aug 13, 2023Updated 3 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Practice your Pyspark skills!☆113Oct 22, 2021Updated 4 years ago
- Docker with Airflow and Spark standalone cluster☆264Aug 5, 2023Updated 3 years ago
- Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼☆46,105Sep 15, 2026Updated 3 weeks ago
- Source code for the "Coding in Motion" series.☆27Sep 21, 2022Updated 4 years ago
- An Awesome List of Open-Source Data Engineering Projects☆3,301Oct 4, 2024Updated 2 years ago
- ☆216Aug 13, 2026Updated last month
- An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS ap…☆27Dec 7, 2022Updated 3 years ago