End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API, sends the data to Kafka, and processes it with Spark before writing to Cassandra. The pipeline, built with Python and Apache Zookeeper, is containerized with Docker for easy deployment and scalability.
☆23Sep 23, 2026Updated last month
Alternatives and similar repositories for e2e-structured-streaming
Users that are interested in e2e-structured-streaming are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Fully dockerized Data Warehouse (DWH) using Airflow, dbt, PostgreSQL and dashboard using redash☆27Nov 12, 2022Updated 4 years ago
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆51Dec 11, 2023Updated 2 years ago
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆25May 14, 2022Updated 4 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆122Jan 8, 2026Updated 10 months ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technolog…☆13Jun 26, 2022Updated 4 years ago
- Autonomous UAV-Based Maritime SAR using onboard YOLOv8/v11 for real-time swimmer detection and precision life buoy deployment. Combines G…☆17Aug 7, 2026Updated 3 months ago
- ☆12Mar 6, 2021Updated 5 years ago
- In this article, you will learn how to set up a real-time data processing and analytics environment using Docker, MySQL, Redpanda, MinIO,…☆11Jun 27, 2023Updated 3 years ago
- Open Data Stack Platform: a collection of projects and pipelines built with open data stack tools for scalable, observable data platform…☆24Sep 19, 2026Updated 2 months ago
- ☆13Sep 15, 2024Updated 2 years ago
- A data pipeline moving data from a Relational database system (RDBMS) to a Hadoop file system (HDFS).☆15Jun 3, 2021Updated 5 years ago
- Building a Comprehensive Repository of Hromada-Level Data in Ukraine to Facilitate Research and Informed Policy Decisions. This repositor…☆17Nov 6, 2026Updated 2 weeks ago
- Template for Retreival-Augmented Generation (RAG) application using Milvus for vector storage, LangGraph for ReAct agentic workflows, and…☆12May 21, 2025Updated last year
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- SQL Tutorials using Jupyter Notebook☆17Apr 9, 2023Updated 3 years ago
- An end-to-end AWS data lakehouse leveraging batch data processing, big data technologies, and advanced business analytics.☆11Feb 12, 2026Updated 9 months ago
- This is a simple object detection application, which can be used to detect military objects☆15Nov 16, 2020Updated 6 years ago
- ☆10Jul 19, 2020Updated 6 years ago
- ETL (Extract, Transform and Load) with the Spark Python API (PySpark) and Hadoop Distributed File System (HDFS)☆17Dec 18, 2018Updated 7 years ago
- End-to-End ELT data pipeline with Postgres, Airbyte, dbt, Dagster, Snowflake and Metabase☆12Jul 13, 2023Updated 3 years ago
- ☆15Jun 2, 2025Updated last year
- Pixel-perfect 24px outline icons for SVG, React, React Native, Vue, Svelte, Solid, Angular, Astro, and Preact.☆14Nov 13, 2026Updated last week
- Medium article "Python package management with uv for dockerized environments"☆14Feb 17, 2025Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Set of tools helping developers to build TEE-enabled application quicker.☆16Aug 22, 2025Updated last year
- ☆12Apr 3, 2026Updated 7 months ago
- NSCollectionView sample for OS X 10.11 ElCapitan☆12Nov 24, 2017Updated 8 years ago
- Big data cluster for training/teaching purposes☆15May 5, 2026Updated 6 months ago
- Spark Notebook docker image☆10Dec 29, 2017Updated 8 years ago
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆13Aug 26, 2023Updated 3 years ago
- A Retrieval-Augmented Generation (RAG) based chatbot system for e-commerce product search and order management. The system provides seman…☆15Nov 25, 2025Updated 11 months ago
- Semantic subtitle aligner and merger for bilingual subtitle syncing.☆15Oct 10, 2026Updated last month
- TTS utility☆12Aug 2, 2020Updated 6 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆53Dec 4, 2023Updated 2 years ago
- City-GAN is a Generative Adversarial Neural Network that generates typical facades of various cities around the world.☆20Mar 22, 2022Updated 4 years ago
- My final project for the Data Engineering Zoomcamp by DataTalksClub.☆10Apr 6, 2023Updated 3 years ago
- A platform that helps developers to better understand CSS through declaration interpretation and may even improve them through suggestion…☆14Jul 3, 2021Updated 5 years ago
- velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docke…☆20Aug 12, 2025Updated last year
- This repository contains an end-to-end data engineering project using Apache Flink, focused on performing sales analytics. The project de…☆12Nov 18, 2023Updated 3 years ago
- Use YOLOv8 with a DJI Tello Drone! Controlled with the Tello app and provides video results post-flight.☆20Dec 8, 2023Updated 2 years ago
Are these results useful?