End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API, sends the data to Kafka, and processes it with Spark before writing to Cassandra. The pipeline, built with Python and Apache Zookeeper, is containerized with Docker for easy deployment and scalability.
☆23Sep 23, 2026Updated last month
Alternatives and similar repositories for e2e-structured-streaming
Users that are interested in e2e-structured-streaming are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- used Airflow, Postgres, Kafka, Spark, and Cassandra, and GitHub Actions to establish an end-to-end data pipeline☆33Oct 25, 2023Updated 3 years ago
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆50Dec 11, 2023Updated 2 years ago
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆25May 14, 2022Updated 4 years ago
- The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technolog…☆13Jun 26, 2022Updated 4 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Open Data Stack Platform: a collection of projects and pipelines built with open data stack tools for scalable, observable data platform…☆23Sep 19, 2026Updated last month
- ☆12Mar 6, 2021Updated 5 years ago
- A demonstration of an ELT (Extract, Load, Transform) pipeline☆32Feb 19, 2024Updated 2 years ago
- Building Data Lakehouse by open source technology. Support end to end data pipeline, from source data on AWS S3 to Lakehouse, visualize a…☆42Dec 15, 2025Updated 10 months ago
- ☆13Sep 15, 2024Updated 2 years ago
- This project aims to move the data from a Relational database system (RDBMS) to a Hadoop file system (HDFS)☆11Apr 29, 2022Updated 4 years ago
- A data pipeline moving data from a Relational database system (RDBMS) to a Hadoop file system (HDFS).☆15Jun 3, 2021Updated 5 years ago
- Template for Retreival-Augmented Generation (RAG) application using Milvus for vector storage, LangGraph for ReAct agentic workflows, and…☆12May 21, 2025Updated last year
- End-to-End ELT data pipeline with Postgres, Airbyte, dbt, Dagster, Snowflake and Metabase☆12Jul 13, 2023Updated 3 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- ETL (Extract, Transform and Load) with the Spark Python API (PySpark) and Hadoop Distributed File System (HDFS)☆17Dec 18, 2018Updated 7 years ago
- ☆15Jun 2, 2025Updated last year
- An AI-powered system that generates sustainable trip plans and tourism recommendations for Southeast Asia. Built using a Retrieval-Augmen…☆14Oct 8, 2024Updated 2 years ago
- ☆12Oct 24, 2026Updated last week
- A testing ground for Plotly Dash app development including app features and experimenting with dashboard visualizations.☆10Oct 15, 2023Updated 3 years ago
- MLOps Zoomcamp 2023 repository for Module 3☆11Jun 1, 2023Updated 3 years ago
- My personal blog☆12Jan 17, 2017Updated 9 years ago
- Semantic subtitle aligner and merger for bilingual subtitle syncing.☆15Oct 10, 2026Updated 3 weeks ago
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆13Aug 26, 2023Updated 3 years ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Easy-to-run demo of a CDC pipeline using Debezium (Kafka Connect), PostgreSQL, Redpanda, and ClickHouse.☆12Dec 1, 2025Updated 11 months ago
- My final project for the Data Engineering Zoomcamp by DataTalksClub.☆10Apr 6, 2023Updated 3 years ago
- TTS utility☆12Aug 2, 2020Updated 6 years ago
- A platform that helps developers to better understand CSS through declaration interpretation and may even improve them through suggestion…☆14Jul 3, 2021Updated 5 years ago
- velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docke…☆20Aug 12, 2025Updated last year
- This repository contains an end-to-end data engineering project using Apache Flink, focused on performing sales analytics. The project de…☆12Nov 18, 2023Updated 2 years ago
- Fontstacks to serve with openmaptiles☆12Apr 18, 2020Updated 6 years ago
- A structured streaming was applied to the robot data from ROS-Gazebo simulation environment using Apache Spark. Data is collected in Kafk…☆19Feb 6, 2022Updated 4 years ago
- ☆11Jan 31, 2019Updated 7 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆47Jan 4, 2024Updated 2 years ago
- Go based task runner☆11Sep 26, 2022Updated 4 years ago
- ☆17Nov 27, 2025Updated 11 months ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆122Jan 8, 2026Updated 9 months ago
- ☆12Oct 2, 2020Updated 6 years ago
- ☆23Jul 8, 2025Updated last year
- ☆13Sep 23, 2023Updated 3 years ago