This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging the data, filling the data warehouse, and running checks on the data quality as the final step. Automate the ETL pipeline and creation of data warehouse using Apache Airflow. Skills include: Using Airflow to …
☆104Aug 11, 2019Updated 6 years ago
Alternatives and similar repositories for Data-Pipelines-with-Airflow
Users that are interested in Data-Pipelines-with-Airflow are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆16Dec 4, 2017Updated 8 years ago
- Skeleton project for Apache Airflow training participants to work on.☆17Apr 13, 2026Updated 3 months ago
- Code for Data Pipelines with Apache Airflow☆824Aug 15, 2024Updated last year
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Nov 22, 2021Updated 4 years ago
- Beginner data engineering project - batch edition☆584Apr 13, 2026Updated 3 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆352Jan 12, 2022Updated 4 years ago
- A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Doc…☆25Nov 19, 2024Updated last year
- Airflow Examples: code samples for Medium articles☆14Jan 10, 2021Updated 5 years ago
- Using Apache Airflow to author, run and monitor complex data pipelines.☆12Oct 24, 2018Updated 7 years ago
- Example end to end data engineering project.☆1,418Dec 8, 2022Updated 3 years ago
- A demonstration of an ELT (Extract, Load, Transform) pipeline☆32Feb 19, 2024Updated 2 years ago
- Desarrollé un proyecto de ETL sobre archivos de diferentes orígenes (CSV, JSON). Luego, utilicé FastAPI para crear una API que permita re…☆10Dec 9, 2022Updated 3 years ago
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- Big Data Engineering practice project, including ETL with Airflow and Spark using AWS S3 and EMR☆92Jul 17, 2019Updated 7 years ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract d…☆24Nov 22, 2021Updated 4 years ago
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆168Jun 16, 2020Updated 6 years ago
- Example of an ETL Pipeline using Airflow☆39Aug 30, 2017Updated 8 years ago
- Solution to Data at ANZ virtual internship on Forage☆10May 30, 2021Updated 5 years ago
- ETL best practices with airflow, with examples☆1,354Sep 25, 2024Updated last year
- Role-Based Access Control System for Python implements ANSI INCITS 359☆15Aug 29, 2025Updated 10 months ago
- This repo is for the Linkedin Learning course: End-to-End Data Engineering Project☆36Nov 9, 2023Updated 2 years ago
- End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API…☆21Jul 26, 2024Updated 2 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆116Jan 8, 2026Updated 6 months ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- A small repository of projects built in my course, REST APIs with Flask and Python.☆22Dec 19, 2017Updated 8 years ago
- Resources and projects from Udacity Data Engineering with AWS nano degree programme☆30Apr 12, 2023Updated 3 years ago
- Personal Data Engineering Projects☆1,024Feb 8, 2023Updated 3 years ago
- ☆31Jul 7, 2023Updated 3 years ago
- streaming eight subreddits from reddit api using kafka producer & spark structured streaming.☆19Jul 1, 2026Updated 3 weeks ago
- This data project can be used as a take-home assignment to learn Pyspark and Data Engineering.☆20Feb 19, 2023Updated 3 years ago
- A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!☆892Apr 16, 2022Updated 4 years ago
- Project - Data Processing and Analysis in Python Course☆39Oct 10, 2018Updated 7 years ago
- System design patterns for machine learning☆20Nov 18, 2020Updated 5 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- This is the git repo for the tutorials I have done for Time Series Forecasting by Jason Brownlee☆11Dec 5, 2018Updated 7 years ago
- Apache Airflow tutorial☆976Nov 3, 2022Updated 3 years ago
- Use Airflow to pull in remote data via API, pub/sub, kinesis, s3 etc. and then store it in s3 for later consumption by other services.☆13Mar 14, 2022Updated 4 years ago
- ☆45Apr 21, 2022Updated 4 years ago
- Implementing best practices for PySpark ETL jobs and applications.☆2,121Jan 1, 2023Updated 3 years ago
- Content for a talk on "The wonderful world of data quality tools in Python"☆18May 5, 2021Updated 5 years ago
- AWS ParallelCluster Module for Terraform☆13Jul 7, 2026Updated 3 weeks ago