End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API, sends the data to Kafka, and processes it with Spark before writing to Cassandra. The pipeline, built with Python and Apache Zookeeper, is containerized with Docker for easy deployment and scalability.
☆21Jul 26, 2024Updated last year
Alternatives and similar repositories for e2e-structured-streaming
Users that are interested in e2e-structured-streaming are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- Business Intelligence and Data Warehousing Project☆14Dec 4, 2019Updated 6 years ago
- Products Information Portal and Microservices☆13May 20, 2026Updated last month
- Open Data Stack Platform: a collection of projects and pipelines built with open data stack tools for scalable, observable data platform…☆22May 11, 2026Updated last month
- The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technolog…☆13Jun 26, 2022Updated 4 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- SQL Tutorials using Jupyter Notebook☆17Apr 9, 2023Updated 3 years ago
- used Airflow, Postgres, Kafka, Spark, and Cassandra, and GitHub Actions to establish an end-to-end data pipeline☆32Oct 25, 2023Updated 2 years ago
- This is a demo project to compare two web scrapping frameworks, Playwright and Selenium and using the new Pipelining tool Dagster☆15Sep 9, 2021Updated 4 years ago
- ☆13Sep 15, 2024Updated last year
- This repo will guide you step-by-step method to create star schema dimensional model.☆25Jun 1, 2021Updated 5 years ago
- A curated list of awesome Python frameworks, libraries, software and resources☆15Jun 6, 2018Updated 8 years ago
- Cutting-edge, opinionated, and ambitious project builder for power users and researchers.☆16Feb 2, 2026Updated 5 months ago
- An Objective-C library for uploading shots to Dribbble.☆13Mar 27, 2012Updated 14 years ago
- Upload shots to dribbble.com☆14Mar 27, 2012Updated 14 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Đồ án tốt nghiệp | Data Lakehouse☆46Feb 9, 2026Updated 4 months ago
- A testing ground for Plotly Dash app development including app features and experimenting with dashboard visualizations.☆10Oct 15, 2023Updated 2 years ago
- ☆69Sep 24, 2025Updated 9 months ago
- NSCollectionView sample for OS X 10.11 ElCapitan☆12Nov 24, 2017Updated 8 years ago
- ☆10Feb 2, 2024Updated 2 years ago
- Underlying package for the 10-line cta☆15Updated this week
- ☆10Jul 19, 2020Updated 5 years ago
- ☆10Aug 20, 2024Updated last year
- ☆13Sep 23, 2023Updated 2 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- A platform that helps developers to better understand CSS through declaration interpretation and may even improve them through suggestion…☆14Jul 3, 2021Updated 5 years ago
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆45Jan 4, 2024Updated 2 years ago
- Modern GIS Web Client for JavaScript, based on MapboxGL-JS, OpenLayers, Leaflet☆13Sep 16, 2022Updated 3 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆115Jan 8, 2026Updated 5 months ago
- TTS utility☆12Aug 2, 2020Updated 5 years ago
- um its my portfolio?☆16Jun 15, 2026Updated 2 weeks ago
- View data on a tile38 server☆14Aug 18, 2024Updated last year
- ☆17Nov 27, 2025Updated 7 months ago
- An example of a project generated with cookiecutter-uv☆16Apr 10, 2026Updated 2 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO☆65Jul 21, 2023Updated 2 years ago
- End-to-end data platform: A PoC Data Platform project utilizing modern data stack (Spark, Airflow, DBT, Trino, Lightdash, Hive metastore,…☆49Oct 14, 2024Updated last year
- ☆11Jan 31, 2019Updated 7 years ago
- [SC2023] POMELO: Fine-grained Population Mapping from Coarse Census Counts and Open Geodata☆13Aug 5, 2024Updated last year
- This repository contains an end-to-end data engineering project using Apache Flink, focused on performing sales analytics. The project de…☆12Nov 18, 2023Updated 2 years ago
- package for snow science data, providing streamlined access to satellite imagery (Sentinel-1/2, HLS, MODIS, etc), weather station data, c…☆15Jun 22, 2026Updated last week
- code and demo for hierarchical stacking paper☆10May 13, 2021Updated 5 years ago