An end-to-end, containerized data pipeline for near-real-time user event analytics using Kafka, ClickHouse, Airflow, and PySpark. Made to learn some common data engineering practices.
☆81Sep 12, 2025Updated 10 months ago
Alternatives and similar repositories for End2End-Data-Pipeline
Users that are interested in End2End-Data-Pipeline are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Project utilising data from the Age of Empires api at 'https://aoestats.io'☆53Dec 8, 2024Updated last year
- A repository defining a simple data pipleine for ETL jobs relating to media metadata.☆53Mar 11, 2025Updated last year
- ☆22Sep 22, 2025Updated 10 months ago
- ☆27Dec 1, 2023Updated 2 years ago
- ☆15Apr 16, 2026Updated 3 months ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Dockerized analytics pipeline where Airflow moves mock-API data into Postgres, runs dbt transformations, and serves clean tables for Powe…☆81Oct 18, 2025Updated 9 months ago
- This project is built for personal purposes. Currently, I am enahancing it with an interactive bot that can recommend and help you improv…☆26Updated this week
- Repo for learning DBT with Snowflake, featuring projects and models for data transformation and automation☆26Mar 31, 2025Updated last year
- ☆23Feb 5, 2024Updated 2 years ago
- ☆20Jan 2, 2026Updated 6 months ago
- 🟣 Data Engineer interview questions and answers to help you prepare for your next machine learning and data science interview in 2026.☆98Jan 4, 2026Updated 6 months ago
- Đồ án tốt nghiệp | Data Lakehouse☆47Feb 9, 2026Updated 5 months ago
- This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data wareh…☆227Oct 23, 2023Updated 2 years ago
- An ETL pipeline that extracts data from S3, stages them in Redshift, and transforms data into a set of dimensional tables☆16May 5, 2020Updated 6 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- ☆23Feb 26, 2026Updated 5 months ago
- Spam Reviews Detection on Vietnamese E-commerce website☆16Jan 28, 2026Updated 6 months ago
- On-premises ELT Pipeline☆32Jul 10, 2025Updated last year
- MLOps Implementation for Disaster Tweets Classifier Application☆24Mar 24, 2024Updated 2 years ago
- Demo of using Airflow☆11Jun 24, 2022Updated 4 years ago
- Code for DE101 book at https://de101.startdataengineering.com/☆120Feb 22, 2026Updated 5 months ago
- End-to-end real-time analytics pipeline with Kafka, Spark micro-batch/Spark Streaming/Flink, ClickHouse, and live dashboards.☆19Mar 2, 2026Updated 4 months ago
- a tool deployed on AWS Lambda to help checking internship slot (HCMUT), Public version☆10May 26, 2025Updated last year
- by Roxs☆25Feb 20, 2026Updated 5 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Create a new GitHub Discussion with GitHub Actions☆17Oct 27, 2025Updated 9 months ago
- Showcasing AI Data Modeling with Lineage & impact analysis on OpenMetadata☆21Apr 16, 2026Updated 3 months ago
- Code for NeurIPS 2024 paper "A SARS-CoV-2 Interaction Dataset and VHH Sequence Corpus for Antibody Language Models"☆16Oct 17, 2024Updated last year
- A sample enterprise application for restaurant business developed in May 2014 using ASP.NET MVC 5, WinForm, WCF, EF, and SQL Server☆10Aug 31, 2022Updated 3 years ago
- Data Vault 2.0: Code generation, Vertica, Airflow☆12Nov 20, 2019Updated 6 years ago
- Variable Selection Network with PyTorch☆12May 29, 2024Updated 2 years ago
- Repo for my book "Data Modeling with Power BI Desktop" (https://www.oreilly.com/library/view/data-modeling-with/9781098148546/)☆24Nov 26, 2025Updated 8 months ago
- End to End RAG LLM AI Assistant using LangChain, Llama3, Gemma2, OpenAI, FlaskAPI, Grafana☆11Nov 24, 2025Updated 8 months ago
- Học Docker trên Ubuntu☆17Jul 16, 2025Updated last year
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- by Roxs☆12May 2, 2025Updated last year
- 🔍 Search Engine from scratch — full-text search, indexing, ranking & real-world patterns using Elasticsearch / Typesense for backend dev…☆18Apr 9, 2026Updated 3 months ago
- Reverse-proxy NGinx doing Basic Authentication, passing Auth user through Websocket headers and Streamlit extracting auth login from webs…☆16Jan 11, 2023Updated 3 years ago
- ☆47Nov 7, 2023Updated 2 years ago
- ☆20Nov 27, 2025Updated 8 months ago
- Sentiment analysis with Vietnamese reviews from Shopee online market☆15Nov 21, 2022Updated 3 years ago
- ☆17Oct 30, 2022Updated 3 years ago