End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API, sends the data to Kafka, and processes it with Spark before writing to Cassandra. The pipeline, built with Python and Apache Zookeeper, is containerized with Docker for easy deployment and scalability.
☆23Jul 26, 2024Updated 2 years ago
Alternatives and similar repositories for e2e-structured-streaming
Users that are interested in e2e-structured-streaming are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆50Dec 11, 2023Updated 2 years ago
- Fully dockerized Data Warehouse (DWH) using Airflow, dbt, PostgreSQL and dashboard using redash☆26Nov 12, 2022Updated 3 years ago
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- End-to-End BI & DW project: Data Warehousing design and modeling (MySQL), ETL (PDI) and Dashboard (Tableau)☆18Aug 10, 2020Updated 6 years ago
- The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technolog…☆13Jun 26, 2022Updated 4 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Open Data Stack Platform: a collection of projects and pipelines built with open data stack tools for scalable, observable data platform…☆23Updated this week
- SQL Tutorials using Jupyter Notebook☆17Apr 9, 2023Updated 3 years ago
- used Airflow, Postgres, Kafka, Spark, and Cassandra, and GitHub Actions to establish an end-to-end data pipeline☆32Oct 25, 2023Updated 2 years ago
- ☆13Sep 15, 2024Updated 2 years ago
- This is a demo project to compare two web scrapping frameworks, Playwright and Selenium and using the new Pipelining tool Dagster☆15Sep 9, 2021Updated 5 years ago
- A demonstration of an ELT (Extract, Load, Transform) pipeline☆32Feb 19, 2024Updated 2 years ago
- This repo will guide you step-by-step method to create star schema dimensional model.☆25Jun 1, 2021Updated 5 years ago
- This project aims to move the data from a Relational database system (RDBMS) to a Hadoop file system (HDFS)☆11Apr 29, 2022Updated 4 years ago
- A data pipeline moving data from a Relational database system (RDBMS) to a Hadoop file system (HDFS).☆15Jun 3, 2021Updated 5 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- TTS utility☆12Aug 2, 2020Updated 6 years ago
- ☆11Aug 20, 2024Updated 2 years ago
- An example of a project generated with cookiecutter-uv☆17Apr 10, 2026Updated 5 months ago
- Building blocks for trend-following CTA strategies: scale-free oscillators, volatility-adjusted returns, matrix shrinkage and a position …☆17Updated this week
- This repository contains an end-to-end data engineering project using Apache Flink, focused on performing sales analytics. The project de…☆12Nov 18, 2023Updated 2 years ago
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆46Jan 4, 2024Updated 2 years ago
- It is a assemble to include all Practice Projects about Big Data Topic, includes Hadoop, Spark, Spark Streaming and Kafka☆11Mar 7, 2019Updated 7 years ago
- A platform that helps developers to better understand CSS through declaration interpretation and may even improve them through suggestion…☆14Jul 3, 2021Updated 5 years ago
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆13Aug 26, 2023Updated 3 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Modern GIS Web Client for JavaScript, based on MapboxGL-JS, OpenLayers, Leaflet☆12Sep 16, 2022Updated 4 years ago
- ☆13Sep 23, 2023Updated 2 years ago
- ☆16Updated this week
- Spark Notebook docker image☆10Dec 29, 2017Updated 8 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆120Jan 8, 2026Updated 8 months ago
- [SC2023] POMELO: Fine-grained Population Mapping from Coarse Census Counts and Open Geodata☆13Aug 5, 2024Updated 2 years ago
- ☆17Nov 27, 2025Updated 9 months ago
- An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS ap…☆27Dec 7, 2022Updated 3 years ago
- Chroma maintenance CLI☆19Aug 15, 2025Updated last year
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Stock Advisor☆12Jun 13, 2025Updated last year
- This is end-to-end-recommender-system repo has full recommender system implementation from collecting data, modeling and deploying machin…☆13May 19, 2021Updated 5 years ago
- package for snow science data, providing streamlined access to satellite imagery (Sentinel-1/2, HLS, MODIS, etc), weather station data, c…☆15Updated this week
- ☆11Jan 31, 2019Updated 7 years ago
- code and demo for hierarchical stacking paper☆10May 13, 2021Updated 5 years ago
- Transformer Conformal Prediction for Time Series☆19Apr 13, 2026Updated 5 months ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆25May 14, 2022Updated 4 years ago