End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API, sends the data to Kafka, and processes it with Spark before writing to Cassandra. The pipeline, built with Python and Apache Zookeeper, is containerized with Docker for easy deployment and scalability.
☆22Jul 26, 2024Updated 2 years ago
Alternatives and similar repositories for e2e-structured-streaming
Users that are interested in e2e-structured-streaming are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- End-to-End BI & DW project: Data Warehousing design and modeling (MySQL), ETL (PDI) and Dashboard (Tableau)☆18Aug 10, 2020Updated 6 years ago
- The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technolog…☆13Jun 26, 2022Updated 4 years ago
- SQL Tutorials using Jupyter Notebook☆17Apr 9, 2023Updated 3 years ago
- used Airflow, Postgres, Kafka, Spark, and Cassandra, and GitHub Actions to establish an end-to-end data pipeline☆32Oct 25, 2023Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ☆13Sep 15, 2024Updated last year
- A demonstration of an ELT (Extract, Load, Transform) pipeline☆32Feb 19, 2024Updated 2 years ago
- ☆12Mar 6, 2021Updated 5 years ago
- This project aims to move the data from a Relational database system (RDBMS) to a Hadoop file system (HDFS)☆11Apr 29, 2022Updated 4 years ago
- A data pipeline moving data from a Relational database system (RDBMS) to a Hadoop file system (HDFS).☆15Jun 3, 2021Updated 5 years ago
- TTS utility☆12Aug 2, 2020Updated 6 years ago
- ☆15Jun 2, 2025Updated last year
- This repository contains an end-to-end data engineering project using Apache Flink, focused on performing sales analytics. The project de…☆12Nov 18, 2023Updated 2 years ago
- ☆10Jul 19, 2020Updated 6 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆45Jan 4, 2024Updated 2 years ago
- It is a assemble to include all Practice Projects about Big Data Topic, includes Hadoop, Spark, Spark Streaming and Kafka☆11Mar 7, 2019Updated 7 years ago
- A platform that helps developers to better understand CSS through declaration interpretation and may even improve them through suggestion…☆14Jul 3, 2021Updated 5 years ago
- Modern GIS Web Client for JavaScript, based on MapboxGL-JS, OpenLayers, Leaflet☆12Sep 16, 2022Updated 3 years ago
- ☆16Feb 11, 2026Updated 6 months ago
- ☆23Jul 8, 2025Updated last year
- Spark Notebook docker image☆10Dec 29, 2017Updated 8 years ago
- View data on a tile38 server☆14Aug 18, 2024Updated last year
- ☆21Jun 10, 2023Updated 3 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- [SC2023] POMELO: Fine-grained Population Mapping from Coarse Census Counts and Open Geodata☆13Aug 5, 2024Updated 2 years ago
- An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS ap…☆27Dec 7, 2022Updated 3 years ago
- Stock Advisor☆12Jun 13, 2025Updated last year
- This is end-to-end-recommender-system repo has full recommender system implementation from collecting data, modeling and deploying machin…☆13May 19, 2021Updated 5 years ago
- ☆11Jan 31, 2019Updated 7 years ago
- I will share DSA notes and code here☆19Mar 24, 2023Updated 3 years ago
- DuckDB Copilot Extension☆10Jan 12, 2026Updated 7 months ago
- A Proxy service using FastAPI and Protocol Buffers (Proto3)☆13Jun 17, 2023Updated 3 years ago
- ☆28Aug 28, 2023Updated 2 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Kafka Connect: How to create a real time data pipeline using Change Data Capture (CDC)☆13Jan 24, 2021Updated 5 years ago
- Sample for using the Elasticsearch Vector Tiles Search API☆11Apr 29, 2022Updated 4 years ago
- Execution of DBT models using Apache Airflow through Docker Compose☆133Jan 3, 2023Updated 3 years ago
- Awesome lists link about all kinds of topics☆19Aug 3, 2026Updated last week
- This repo demonstrates an Apache Arrow Flight server implementation in Kubernetes.☆12Oct 25, 2024Updated last year
- A jupyter kernel for GoPlus☆23Mar 4, 2021Updated 5 years ago
- ☆12Oct 2, 2020Updated 5 years ago