📡 Real-time data pipeline with Kafka, Flink, Iceberg, Trino, MinIO, and Superset. Ideal for learning data systems.
☆81Jan 18, 2025Updated last year
Alternatives and similar repositories for real-time-data-pipeline
Users that are interested in real-time-data-pipeline are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- A complete data engineering project demonstrating modern data stack practices with Apache Flink, Iceberg, Trino and Superset☆27Sep 29, 2025Updated last year
- Session slides & demonstration code from the "Custom Admin Monitoring for Microsoft Fabric & Power BI" session☆12Apr 29, 2025Updated last year
- Local Environment to Practice Data Engineering☆143Dec 30, 2024Updated last year
- An AWS Data Engineering End-to-End Project (Glue, Lambda, Kinesis, Redshift, QuickSight, Athena, EC2, S3)☆17Sep 20, 2023Updated 3 years ago
- Stream processing guidelines and examples using Apache Flink and Apache Spark☆47Apr 21, 2023Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ☆67Updated this week
- Data Engineering Youtube Project☆12Jun 29, 2023Updated 3 years ago
- A complete real-time Change Data Capture (CDC) pipeline using Apache Flink, MariaDB, and Docker Compose. This project demonstrates how to…☆38Updated this week
- Example GitHub Actions for Apache Kafka client application development for local and Confluent Cloud☆15Aug 1, 2022Updated 4 years ago
- ☆21Mar 31, 2024Updated 2 years ago
- This project contain build end-to-end e-commerce data from data source into data warehouse and visualization.☆13Sep 5, 2024Updated 2 years ago
- A practical demo using Atomic Agents, showing how to build your own code generation agent that actually executes the code it writes in a …☆14Nov 24, 2024Updated last year
- Building Data Lakehouse by open source technology. Support end to end data pipeline, from source data on AWS S3 to Lakehouse, visualize a…☆42Dec 15, 2025Updated 10 months ago
- Handy yt-dlp wrapper to download Youtube videos transcripts☆17Sep 29, 2026Updated last month
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- MLOps Zoomcamp 2023 repository for Module 3☆11Jun 1, 2023Updated 3 years ago
- Data Engineering Projects☆11Mar 30, 2024Updated 2 years ago
- ☆27May 17, 2026Updated 5 months ago
- Conduktor Helm Chart Repository☆11Oct 23, 2026Updated 2 weeks ago
- High-performance LLM query cache with semantic search. Reduce API costs 80% and latency from 8.5s to 1ms using Redis + Qdrant vector DB…☆13Dec 2, 2025Updated 11 months ago
- ☆11Oct 10, 2018Updated 8 years ago
- An e2e pipeline using dlt, dagster, duckdb, and dbt-core☆23Mar 27, 2026Updated 7 months ago
- Repository for Reference for Apache Iceberg LinkedIN Learning Courses☆17Jan 27, 2025Updated last year
- This solution sets up an automated migration process for moving tables (Apache Iceberg and Hive tables) registered in AWS Glue Table Cata…☆13Aug 8, 2025Updated last year
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- Run an open-source data LakeHouse locally using Docker Compose☆12May 31, 2024Updated 2 years ago
- Demo - API Observability with Tyk, OpenTelemetry, Jaeger and Grafana☆11Apr 22, 2026Updated 6 months ago
- 🦆 Batch data pipeline with Airflow, DuckDB, Delta Lake, Trino, MinIO, and Metabase. Full observability and data quality.☆88Nov 5, 2025Updated last year
- A sample implementation of stream writes to an Iceberg table on GCS using Flink and reading it using Trino☆22May 30, 2022Updated 4 years ago
- An advanced RAG (Retrieval-Augmented Generation) system that provides intelligent access to Cerebras inference documentation with citatio…☆11May 24, 2025Updated last year
- learning-by-doing data model built with dbt-core☆17Oct 26, 2026Updated last week
- TMI 2023: FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognition☆13Mar 19, 2024Updated 2 years ago
- Study Guide☆11Jan 31, 2025Updated last year
- Simple project using pyflink, kafka and postgre containerized using Docker☆12Aug 26, 2024Updated 2 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- Files for the Docker and Kubernetes on Google Cloud Hands-On labs☆11Mar 14, 2023Updated 3 years ago
- In this project I have built etl pipline which scraps the trending repository based on month,week and day LIVE extract other related info…☆12Sep 9, 2023Updated 3 years ago
- Create an ECS environment for a distributed Dagster installation☆12Feb 20, 2024Updated 2 years ago
- The Metadata Driven framework for Databricks Lakeflow Declarative Pipelines (formerly Delta Live Tables). Metadata framework that generat…☆67Updated this week
- Azure DP 900 notes☆10Jan 23, 2022Updated 4 years ago
- ☆16Oct 8, 2025Updated last year
- Open Data Stack Platform: a collection of projects and pipelines built with open data stack tools for scalable, observable data platform…☆23Sep 19, 2026Updated last month