This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging the data, filling the data warehouse, and running checks on the data quality as the final step. Automate the ETL pipeline and creation of data warehouse using Apache Airflow. Skills include: Using Airflow to …
☆104Aug 11, 2019Updated 6 years ago
Alternatives and similar repositories for Data-Pipelines-with-Airflow
Users that are interested in Data-Pipelines-with-Airflow are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆16Dec 4, 2017Updated 8 years ago
- ☆17Nov 20, 2020Updated 5 years ago
- Skeleton project for Apache Airflow training participants to work on.☆17Apr 13, 2026Updated 2 months ago
- Code for Data Pipelines with Apache Airflow☆822Aug 15, 2024Updated last year
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Nov 22, 2021Updated 4 years ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- ☆21Jan 13, 2024Updated 2 years ago
- Beginner data engineering project - batch edition☆582Apr 13, 2026Updated 2 months ago
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆350Jan 12, 2022Updated 4 years ago
- Using Apache Airflow to author, run and monitor complex data pipelines.☆12Oct 24, 2018Updated 7 years ago
- Example end to end data engineering project.☆1,417Dec 8, 2022Updated 3 years ago
- Airflow Examples: code samples for Medium articles☆14Jan 10, 2021Updated 5 years ago
- For this project I am creating an ETL (Extract, Transform, and Load) pipeline using Python, RegEx, and SQL Database. The goal is to retri…☆26Feb 9, 2021Updated 5 years ago
- A demonstration of an ELT (Extract, Load, Transform) pipeline☆32Feb 19, 2024Updated 2 years ago
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- Big Data Engineering practice project, including ETL with Airflow and Spark using AWS S3 and EMR☆92Jul 17, 2019Updated 6 years ago
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆167Jun 16, 2020Updated 6 years ago
- Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard. The dashboa…☆270Jan 1, 2023Updated 3 years ago
- This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and …☆34Feb 9, 2024Updated 2 years ago
- ETL best practices with airflow, with examples☆1,355Sep 25, 2024Updated last year
- This project focuses on time series forecasting to predict store sales for Corporation Favorita, a large Ecuadorian-based grocery retaile…☆19Dec 4, 2023Updated 2 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆115Jan 8, 2026Updated 6 months ago
- End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API…☆21Jul 26, 2024Updated last year
- A small repository of projects built in my course, REST APIs with Flask and Python.☆22Dec 19, 2017Updated 8 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Personal Data Engineering Projects☆1,023Feb 8, 2023Updated 3 years ago
- A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!☆885Apr 16, 2022Updated 4 years ago
- Project - Data Processing and Analysis in Python Course☆39Oct 10, 2018Updated 7 years ago
- A PowerBI dashboard to analyze raw sales data from a multinational pharmaceutical manufacturing company and get insights into the perform…☆22Aug 30, 2023Updated 2 years ago
- Apache Airflow tutorial☆976Nov 3, 2022Updated 3 years ago
- An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.☆1,520Mar 9, 2020Updated 6 years ago
- A comprehensive analysis of sales data to gain insights and predict future sales trends, visualized and presented through Power BI dashbo…☆12Jun 20, 2023Updated 3 years ago
- ☆45Apr 21, 2022Updated 4 years ago
- Content for a talk on "The wonderful world of data quality tools in Python"☆18May 5, 2021Updated 5 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Implementing best practices for PySpark ETL jobs and applications.☆2,121Jan 1, 2023Updated 3 years ago
- Sample Airflow DAGs to load data from the CovidTracking API to Snowflake via an AWS S3 intermediary.☆16Jan 6, 2021Updated 5 years ago
- All the code related to building my own data lake☆21May 22, 2023Updated 3 years ago
- Resources for the free AWS Data Engineering course on youtube☆107Aug 30, 2021Updated 4 years ago
- Amazon Redshift offers a common query interface against data stored in fast, local storage as well as data from high-capacity, inexpensiv…☆13Nov 26, 2018Updated 7 years ago
- Productionalizing Data Pipelines with Apache Airflow☆116Jun 18, 2022Updated 4 years ago
- Step by step tutorial for those who have zero knowledge to Amazon EKS☆14Jun 18, 2026Updated 2 weeks ago