yahwang / Awesome-Data-Engineering
📒(GitBook) A curated list of awesome Data Engineering resources
☆33Updated 2 months ago
Related projects: ⓘ
- Playground for Lakehouse (Iceberg, Hudi, Spark, Flink, Trino, DBT, Airflow, Kafka, Debezium CDC)☆40Updated 11 months ago
- Data engineering interviews Q&A for data community by data community☆60Updated 4 years ago
- Design/Implement stream/batch architecture on NYC taxi data | #DE☆26Updated 3 years ago
- A real-time event pipeline around Kafka Ecosystem for Chicago Transit Authority.☆29Updated last year
- ☆35Updated 2 months ago
- Weekly Data Engineering Newsletter☆93Updated 2 months ago
- ☆48Updated 2 years ago
- (project & tutorial) dag pipeline tests + ci/cd setup☆84Updated 3 years ago
- Projects done in the Data Engineer Nanodegree Program by Udacity.com☆83Updated last year
- Mastering Big Data Analytics with PySpark, Published by Packt☆153Updated last month
- Code snippets for Data Engineering Design Patterns book☆27Updated this week
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Updated 2 years ago
- ☆44Updated 2 years ago
- I am using confluent Kafka cluster to produce and consume scraped data. In this project, I've created a real-time data pipeline that uti…☆28Updated last year
- Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO☆56Updated last year
- Spark data pipeline that processes movie ratings data.☆26Updated last month
- Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validatio …☆53Updated last year
- PySpark Cheatsheet☆35Updated last year
- Full stack data engineering tools and infrastructure set-up☆38Updated 3 years ago
- A Snowflake GPT Demo using SqlAlchemy☆23Updated last year
- RedditR for Content Engagement and Recommendation☆20Updated 6 years ago
- A data pipeline moving data from a Relational database system (RDBMS) to a Hadoop file system (HDFS).☆15Updated 3 years ago
- Delta-Lake, ETL, Spark, Airflow☆42Updated last year
- Dockerizing an Apache Spark Standalone Cluster☆43Updated 2 years ago
- Simple stream processing pipeline☆89Updated 3 months ago
- ☆7Updated 2 years ago
- PySpark data-pipeline testing and CICD☆28Updated 3 years ago
- A repository of sample code to show data quality checking best practices using Airflow.☆71Updated last year
- This repo contains commands that data engineers use in day to day work.☆58Updated last year
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆127Updated 4 years ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆20Updated 2 years ago