🦆 Batch data pipeline with Airflow, DuckDB, Delta Lake, Trino, MinIO, and Metabase. Full observability and data quality.
☆85Nov 5, 2025Updated 4 months ago
Alternatives and similar repositories for batch-data-pipeline
Users that are interested in batch-data-pipeline are comparing it to the libraries listed below
Sorting:
- ☆15Mar 29, 2024Updated last year
- Use MobileNet SSD and openCV to detect and count car on road☆12Jan 13, 2020Updated 6 years ago
- Spin up a minimalistic Data Analytics Platform on a European cloud provider☆19Sep 9, 2025Updated 5 months ago
- Hexagonal (ports and adapters) architecture applied to Spark and Python data engineering project☆33Jul 26, 2023Updated 2 years ago
- ☆16Apr 1, 2025Updated 11 months ago
- End-to-end data platform: A PoC Data Platform project utilizing modern data stack (Spark, Airflow, DBT, Trino, Lightdash, Hive metastore,…☆48Oct 14, 2024Updated last year
- EDA, manipulating raw data, drawing conclusions from plots on Netflix data.☆11May 25, 2021Updated 4 years ago
- Real-time Credit card Fraud detection using Spark Streaming, Spark ML, Spark SQL, Kafka, Cassandra and Airflow☆11Jul 1, 2022Updated 3 years ago
- Realistic OLTP data simulator for CDC testing with Debezium☆17Nov 5, 2025Updated 4 months ago
- This is the HTML-CSS source code to build my personal website.☆10Nov 13, 2025Updated 3 months ago
- Interactive web-based dashboard to manage traffic flow using YOLOX, DeepSORT☆10Jul 30, 2022Updated 3 years ago
- Multi-threaded simple proxy server in Python with file caching☆11Oct 4, 2020Updated 5 years ago
- ☆10Jan 27, 2025Updated last year
- An open and introductory book for the Python API of Apache Spark (pyspark) 📚📖☆12Sep 19, 2025Updated 5 months ago
- ☆12Sep 23, 2023Updated 2 years ago
- A testing ground for Plotly Dash app development including app features and experimenting with dashboard visualizations.☆10Oct 15, 2023Updated 2 years ago
- Machine Learning Model and Deployment for Classification of Mango Varieties☆10Dec 22, 2022Updated 3 years ago
- An opinionated theme-aware shell prompt☆18Sep 24, 2025Updated 5 months ago
- Para entender e aprender um pouco sobre o Apache Kafka.https://www.youtube.com/channel/UC3pevgVzUWKo5CoWdhDsoHw☆13Jan 8, 2026Updated last month
- Wind power and energy yield forecast☆11Updated this week
- ☆13Feb 2, 2026Updated last month
- IFC Parser is a Python script for automatical creating material takeoff from a properly exported IFC2X3 file, and converting that to JSON…☆13Oct 12, 2022Updated 3 years ago
- A portable Datamart and Business Intelligence suite built with Docker, Airflow, dbt, duckdb and Superset☆47Dec 13, 2025Updated 2 months ago
- Question and Answer application using AWS Bedrock, AWS ECS, Langchain, Qdrant, and FastAPI☆15Feb 27, 2024Updated 2 years ago
- um its my portfolio?☆16Feb 10, 2026Updated 3 weeks ago
- My NixOs dotfiles☆13Feb 10, 2026Updated 3 weeks ago
- ☆10Jul 20, 2020Updated 5 years ago
- My solutions to exercises from various SQL learning courses and platforms☆12Jun 23, 2021Updated 4 years ago
- ☆11Nov 21, 2023Updated 2 years ago
- TakaTime is a blazingly fast, privacy-focused coding time tracker for Neovim. It works just like WakaTime, but with one major difference…☆54Updated this week
- This repository contains the capstone project carried out as part of Machine Learning Zoomcamp course☆10Dec 26, 2022Updated 3 years ago
- In this repository you'll find Data Science Projects☆10Mar 6, 2024Updated 2 years ago
- kitty-sessionizer provides session management with session resurrection for the kitty terminal☆15Mar 4, 2025Updated last year
- This is a fork of mayTermux's fork of rxfetch. So basically a fork of a fork?☆10May 21, 2022Updated 3 years ago
- Code accompanying the paper "Fighting Class Imbalance with Contrastive Learning" (MICCAI2021)☆10Nov 24, 2021Updated 4 years ago
- This workshop will familiarize you with some of the key steps towards building an autonomous driving data lake and extracting images from…☆10Jul 12, 2022Updated 3 years ago
- Repository for Data Engineering Zoomcamp 2024☆14Mar 25, 2024Updated last year
- ☆11Nov 11, 2024Updated last year
- Miscellaneous codes and writings for MLOps☆15Updated this week