ddgope / Data-Pipelines-with-AirflowView external linksLinks
This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging the data, filling the data warehouse, and running checks on the data quality as the final step. Automate the ETL pipeline and creation of data warehouse using Apache Airflow. Skills include: Using Airflow to …
☆97Aug 11, 2019Updated 6 years ago
Alternatives and similar repositories for Data-Pipelines-with-Airflow
Users that are interested in Data-Pipelines-with-Airflow are comparing it to the libraries listed below
Sorting:
- Creating Data Pipelines with Apache Airflow to manage ETL from Amazon S3 into Amazon Redshift☆14Jun 12, 2019Updated 6 years ago
- ☆16Dec 4, 2017Updated 8 years ago
- ☆17Nov 20, 2020Updated 5 years ago
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Nov 22, 2021Updated 4 years ago
- Code for Data Pipelines with Apache Airflow☆815Aug 15, 2024Updated last year
- Beginner data engineering project - batch edition☆564Jan 22, 2025Updated last year
- Example end to end data engineering project.☆1,384Dec 8, 2022Updated 3 years ago
- ☆11Jan 5, 2023Updated 3 years ago
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆162Jun 16, 2020Updated 5 years ago
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆347Jan 12, 2022Updated 4 years ago
- Face Verification API☆11Sep 27, 2021Updated 4 years ago
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated last year
- ☆44Apr 21, 2022Updated 3 years ago
- ☆21Jan 13, 2024Updated 2 years ago
- Data models, build data warehouses and data lakes, automate data pipelines, and worked with massive datasets.☆13Jul 16, 2019Updated 6 years ago
- Content for a talk on "The wonderful world of data quality tools in Python"☆18May 5, 2021Updated 4 years ago
- ETL best practices with airflow, with examples☆1,353Sep 25, 2024Updated last year
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆17Oct 1, 2019Updated 6 years ago
- A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Doc…☆22Nov 19, 2024Updated last year
- Causal Inference Using Quasi-Experimental Methods☆20Jan 15, 2021Updated 5 years ago
- ☆14Aug 9, 2017Updated 8 years ago
- Resources and projects from Udacity Data Engineering with AWS nano degree programme☆28Apr 12, 2023Updated 2 years ago
- Apache Airflow tutorial☆975Nov 3, 2022Updated 3 years ago
- ☆34Jun 17, 2020Updated 5 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆108Jan 8, 2026Updated last month
- Personal Data Engineering Projects☆989Feb 8, 2023Updated 3 years ago
- Data Engineering Bootcamp☆30Aug 5, 2025Updated 6 months ago
- Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard. The dashboa…☆245Jan 1, 2023Updated 3 years ago
- ☆31Apr 4, 2022Updated 3 years ago
- A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!☆837Apr 16, 2022Updated 3 years ago
- This repo has some proposed agenda for Azure Machine Learning related hands-on workshops.☆11Feb 2, 2021Updated 5 years ago
- ☆10May 25, 2021Updated 4 years ago
- A tutorial on locality sensitive hashing, using MinHashing for document similarity and CosineSimilarity for Euclidean space similarity.☆34Feb 2, 2021Updated 5 years ago
- Simple ETL pipeline using Python☆29May 22, 2023Updated 2 years ago
- ❤ Namaste React Live Course from Zero to Hero 🚀 by Akshay Saini(Founder of NamasteDev). This repository for Assignment & Class Notes tak…☆12Aug 15, 2023Updated 2 years ago
- ☆31Jul 7, 2023Updated 2 years ago
- A template repo for building and releasing Airflow provider packages.☆73Sep 20, 2025Updated 4 months ago
- ☆30Nov 16, 2023Updated 2 years ago
- My lecture notes and assignment solutions for the Coursera machine learning class taught by Andrew Ng.☆34Sep 28, 2017Updated 8 years ago