End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API, sends the data to Kafka, and processes it with Spark before writing to Cassandra. The pipeline, built with Python and Apache Zookeeper, is containerized with Docker for easy deployment and scalability.
☆23Sep 23, 2026Updated 2 weeks ago
Alternatives and similar repositories for e2e-structured-streaming
Users that are interested in e2e-structured-streaming are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆50Dec 11, 2023Updated 2 years ago
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- Open Data Stack Platform: a collection of projects and pipelines built with open data stack tools for scalable, observable data platform…☆23Sep 19, 2026Updated 3 weeks ago
- ☆13Sep 15, 2024Updated 2 years ago
- This is a demo project to compare two web scrapping frameworks, Playwright and Selenium and using the new Pipelining tool Dagster☆15Sep 9, 2021Updated 5 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- A demonstration of an ELT (Extract, Load, Transform) pipeline☆32Feb 19, 2024Updated 2 years ago
- ☆12Mar 6, 2021Updated 5 years ago
- This repo will guide you step-by-step method to create star schema dimensional model.☆25Jun 1, 2021Updated 5 years ago
- A data pipeline moving data from a Relational database system (RDBMS) to a Hadoop file system (HDFS).☆15Jun 3, 2021Updated 5 years ago
- Upload shots to dribbble.com☆14Mar 27, 2012Updated 14 years ago
- TTS utility☆12Aug 2, 2020Updated 6 years ago
- ☆11Aug 20, 2024Updated 2 years ago
- This repository contains an end-to-end data engineering project using Apache Flink, focused on performing sales analytics. The project de…☆12Nov 18, 2023Updated 2 years ago
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆47Jan 4, 2024Updated 2 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- It is a assemble to include all Practice Projects about Big Data Topic, includes Hadoop, Spark, Spark Streaming and Kafka☆11Mar 7, 2019Updated 7 years ago
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆13Aug 26, 2023Updated 3 years ago
- Modern GIS Web Client for JavaScript, based on MapboxGL-JS, OpenLayers, Leaflet☆12Sep 16, 2022Updated 4 years ago
- ☆13Sep 23, 2023Updated 3 years ago
- ☆16Sep 19, 2026Updated 3 weeks ago
- Spark Notebook docker image☆10Dec 29, 2017Updated 8 years ago
- View data on a tile38 server☆14Aug 18, 2024Updated 2 years ago
- ☆21Jun 10, 2023Updated 3 years ago
- Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO☆66Jul 21, 2023Updated 3 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆120Jan 8, 2026Updated 9 months ago
- An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS ap…☆27Dec 7, 2022Updated 3 years ago
- DuckDB DuckLake Demos☆42Jun 1, 2025Updated last year
- 🚀 A simple javascript template for rapid development of GitHub actions.☆17Feb 24, 2023Updated 3 years ago
- Chroma maintenance CLI☆19Aug 15, 2025Updated last year
- Cutting-edge, opinionated, and ambitious project builder for power users and researchers.☆16Jul 22, 2026Updated 2 months ago
- This is end-to-end-recommender-system repo has full recommender system implementation from collecting data, modeling and deploying machin…☆13May 19, 2021Updated 5 years ago
- package for snow science data, providing streamlined access to satellite imagery (Sentinel-1/2, HLS, MODIS, etc), weather station data, c…☆16Updated this week
- ☆11Jan 31, 2019Updated 7 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- I will share DSA notes and code here☆18Mar 24, 2023Updated 3 years ago
- code and demo for hierarchical stacking paper☆10May 13, 2021Updated 5 years ago
- DuckDB Copilot Extension☆10Jan 12, 2026Updated 9 months ago
- A Proxy service using FastAPI and Protocol Buffers (Proto3)☆13Jun 17, 2023Updated 3 years ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆25May 14, 2022Updated 4 years ago
- ☆12Feb 7, 2024Updated 2 years ago
- ☆27Aug 28, 2023Updated 3 years ago