An end-to-end, containerized data pipeline for near-real-time user event analytics using Kafka, ClickHouse, Airflow, and PySpark. Made to learn some common data engineering practices.
☆80Sep 12, 2025Updated last year
Alternatives and similar repositories for End2End-Data-Pipeline
Users that are interested in End2End-Data-Pipeline are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Project utilising data from the Age of Empires api at 'https://aoestats.io'☆53Dec 8, 2024Updated last year
- A repository defining a simple data pipleine for ETL jobs relating to media metadata.☆52Mar 11, 2025Updated last year
- in progress☆23May 19, 2026Updated 5 months ago
- ☆27Dec 1, 2023Updated 2 years ago
- ☆26Jan 31, 2023Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Repo for learning DBT with Snowflake, featuring projects and models for data transformation and automation☆38Sep 20, 2026Updated last month
- This project is built for personal purposes. Currently, I am enahancing it with an interactive bot that can recommend and help you improv…☆26Jul 27, 2026Updated 2 months ago
- Dockerized analytics pipeline where Airflow moves mock-API data into Postgres, runs dbt transformations, and serves clean tables for Powe…☆84Oct 18, 2025Updated last year
- ☆13Jun 14, 2024Updated 2 years ago
- 🟣 Data Engineer interview questions and answers to help you prepare for your next machine learning and data science interview in 2026.☆107Jan 4, 2026Updated 9 months ago
- ☆17Dec 12, 2024Updated last year
- This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data wareh…☆229Oct 23, 2023Updated 3 years ago
- An ETL pipeline that extracts data from S3, stages them in Redshift, and transforms data into a set of dimensional tables☆16May 5, 2020Updated 6 years ago
- ☆23Feb 26, 2026Updated 7 months ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- A minimal docker compose setup for experimenting with cloud agnostic Lakehouse Architectures Apache Spark with Hive Metastore + Delta Lak…☆34Apr 17, 2024Updated 2 years ago
- Spam Reviews Detection on Vietnamese E-commerce website☆18Jan 28, 2026Updated 8 months ago
- ☆19Feb 11, 2026Updated 8 months ago
- Patterns and concepts for building resilient data pipelines in Python and Scala☆17Aug 27, 2024Updated 2 years ago
- Code snippets for Data Engineering Design Patterns book☆433Jun 13, 2026Updated 4 months ago
- ☆278Aug 5, 2026Updated 2 months ago
- Repositorio de estudio para preparar la certificación AWS Cloud Practitioner☆20Jun 5, 2025Updated last year
- Code base for CDE bootcamp☆87Sep 7, 2026Updated last month
- ☆23Feb 8, 2023Updated 3 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.☆18Jan 12, 2022Updated 4 years ago
- MLOps Implementation for Disaster Tweets Classifier Application☆24Mar 24, 2024Updated 2 years ago
- ☆15Jun 21, 2022Updated 4 years ago
- Teaching materials for a MSc Course about LLMs☆15May 31, 2026Updated 4 months ago
- Code for DE101 book at https://de101.startdataengineering.com/☆136Feb 22, 2026Updated 8 months ago
- End-to-end real-time analytics pipeline with Kafka, Spark micro-batch/Spark Streaming/Flink, ClickHouse, and live dashboards.☆21Mar 2, 2026Updated 7 months ago
- Stream processing pipeline from Finnhub websocket using Spark, Kafka, Kubernetes and more☆445Nov 28, 2023Updated 2 years ago
- 🤖 CI/CD pipeline tutorial using GitHub Actions — covering automated build, test, and deployment workflows for modern web applications.☆115Apr 8, 2026Updated 6 months ago
- Data Modeling Guide: ClickHouse ETL with NOAA Weather Data Example☆12Sep 8, 2025Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Cortana Intelligence Solution for Shipping and Distribution Forecasting☆33Dec 16, 2018Updated 7 years ago
- ☆66Apr 2, 2025Updated last year
- Học Docker trên Ubuntu☆17Jul 16, 2025Updated last year
- Data pipeline project☆66Feb 11, 2025Updated last year
- Sentiment analysis with Vietnamese reviews from Shopee online market☆15Nov 21, 2022Updated 3 years ago
- ☆20Nov 27, 2025Updated 10 months ago
- ☆10Jun 1, 2023Updated 3 years ago