KennethanCeyer / awesome-data-pipeline
Awesome list for datapipeline
☆34Updated 2 years ago
Alternatives and similar repositories for awesome-data-pipeline:
Users that are interested in awesome-data-pipeline are comparing it to the libraries listed below
- 📒(GitBook) A curated list of awesome Data Engineering resources☆35Updated 3 weeks ago
- Full stack data engineering tools and infrastructure set-up☆51Updated 4 years ago
- Playground for Lakehouse (Iceberg, Hudi, Spark, Flink, Trino, DBT, Airflow, Kafka, Debezium CDC)☆56Updated last year
- A curated list of awesome Databricks resources, including Spark☆17Updated 9 months ago
- Code snippets for Data Engineering Design Patterns book☆80Updated last month
- Jupyter notebooks and AWS CloudFormation template to show how Hudi, Iceberg, and Delta Lake work☆47Updated 2 years ago
- A modern ELT demo using airbyte, dbt, snowflake and dagster☆27Updated 2 years ago
- A curated list of awesome DataOps tools☆185Updated 6 months ago
- Spark data pipeline that processes movie ratings data.☆28Updated 3 weeks ago
- Apache Spark Guide☆31Updated 3 years ago
- Curated list of resources about Apache Airflow☆19Updated 4 years ago
- Delta Lake Documentation☆49Updated 10 months ago
- A repository of sample code to show data quality checking best practices using Airflow.☆76Updated 2 years ago
- ☆11Updated 5 months ago
- Repo for everything open table formats (Iceberg, Hudi, Delta Lake) and the overall Lakehouse architecture☆60Updated 3 months ago
- A simple Data Engineering solution for testing or education purposes. You only need to know SQL and Python to understand this project. Da…☆25Updated 2 years ago
- Building Data Lakehouse by open source technology. Support end to end data pipeline, from source data on AWS S3 to Lakehouse, visualize a…☆25Updated last year
- For a series of posts on Amazon MSK, Amazon EKS, and Amazon EMR☆66Updated 3 years ago
- ☆17Updated 8 months ago
- A curated list of awesome open source tools and commercial products to catalog, version, and manage data 🚀☆32Updated 3 years ago
- Sample code to collect Apache Iceberg metrics for table monitoring☆26Updated 8 months ago
- This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenario…☆23Updated 5 months ago
- Finance 🏦 Data Builder 🛠️ @ postgres 🐘☆21Updated 4 years ago
- ☆12Updated 3 years ago
- A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects☆219Updated this week
- Data Tools Subjective List☆83Updated last year
- DataHub on AWS demonstration resources☆10Updated 2 years ago
- 📆 Run, schedule, and manage your dbt jobs using Kubernetes.☆24Updated 6 years ago
- ☆46Updated last week
- A portable Datamart and Business Intelligence suite built with Docker, Airflow, dbt, PostgreSQL and Superset☆40Updated 5 months ago