This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging the data, filling the data warehouse, and running checks on the data quality as the final step. Automate the ETL pipeline and creation of data warehouse using Apache Airflow. Skills include: Using Airflow to …
☆100Aug 11, 2019Updated 6 years ago
Alternatives and similar repositories for Data-Pipelines-with-Airflow
Users that are interested in Data-Pipelines-with-Airflow are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆16Dec 4, 2017Updated 8 years ago
- ☆17Nov 20, 2020Updated 5 years ago
- Skeleton project for Apache Airflow training participants to work on.☆17Jul 9, 2020Updated 5 years ago
- Code for Data Pipelines with Apache Airflow☆815Aug 15, 2024Updated last year
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Nov 22, 2021Updated 4 years ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- ☆21Jan 13, 2024Updated 2 years ago
- Beginner data engineering project - batch edition☆577Updated this week
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆349Jan 12, 2022Updated 4 years ago
- Using Apache Airflow to author, run and monitor complex data pipelines.☆12Oct 24, 2018Updated 7 years ago
- Example end to end data engineering project.☆1,404Dec 8, 2022Updated 3 years ago
- For this project I am creating an ETL (Extract, Transform, and Load) pipeline using Python, RegEx, and SQL Database. The goal is to retri…☆26Feb 9, 2021Updated 5 years ago
- A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Doc…☆23Nov 19, 2024Updated last year
- A demonstration of an ELT (Extract, Load, Transform) pipeline☆31Feb 19, 2024Updated 2 years ago
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- Big Data Engineering practice project, including ETL with Airflow and Spark using AWS S3 and EMR☆90Jul 17, 2019Updated 6 years ago
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆168Jun 16, 2020Updated 5 years ago
- This Power BI dashboard project offers a comprehensive analysis of Blinkit, India's Last Minute App, focusing on sales performance, custo…☆16Aug 18, 2024Updated last year
- Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard. The dashboa…☆267Jan 1, 2023Updated 3 years ago
- This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and …☆34Feb 9, 2024Updated 2 years ago
- Solution to Data at ANZ virtual internship on Forage☆10May 30, 2021Updated 4 years ago
- Example of an ETL Pipeline using Airflow☆39Aug 30, 2017Updated 8 years ago
- ETL best practices with airflow, with examples☆1,352Sep 25, 2024Updated last year
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆111Jan 8, 2026Updated 3 months ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API…☆21Jul 26, 2024Updated last year
- This repo is for the Linkedin Learning course: End-to-End Data Engineering Project☆31Nov 9, 2023Updated 2 years ago
- streaming eight subreddits from reddit api using kafka producer & spark structured streaming.☆19Apr 5, 2026Updated 2 weeks ago
- A small repository of projects built in my course, REST APIs with Flask and Python.☆22Dec 19, 2017Updated 8 years ago
- Personal Data Engineering Projects☆1,005Feb 8, 2023Updated 3 years ago
- ☆31Jul 7, 2023Updated 2 years ago
- A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!☆872Apr 16, 2022Updated 4 years ago
- Built a Data Pipeline for a Retail store using AWS services that collects data from its transactional database (OLTP) in Snowflake and tr…☆12May 25, 2023Updated 2 years ago
- System design patterns for machine learning☆20Nov 18, 2020Updated 5 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Project - Data Processing and Analysis in Python Course☆39Oct 10, 2018Updated 7 years ago
- A PowerBI dashboard to analyze raw sales data from a multinational pharmaceutical manufacturing company and get insights into the perform…☆17Aug 30, 2023Updated 2 years ago
- Apache Airflow tutorial☆974Nov 3, 2022Updated 3 years ago
- An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.☆1,505Mar 9, 2020Updated 6 years ago
- A comprehensive analysis of sales data to gain insights and predict future sales trends, visualized and presented through Power BI dashbo…☆12Jun 20, 2023Updated 2 years ago
- ☆12Jan 20, 2023Updated 3 years ago
- Desarrollé un proyecto de ETL sobre archivos de diferentes orígenes (CSV, JSON). Luego, utilicé FastAPI para crear una API que permita re…☆10Dec 9, 2022Updated 3 years ago