This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging the data, filling the data warehouse, and running checks on the data quality as the final step. Automate the ETL pipeline and creation of data warehouse using Apache Airflow. Skills include: Using Airflow to …
☆104Aug 11, 2019Updated 7 years ago
Alternatives and similar repositories for Data-Pipelines-with-Airflow
Users that are interested in Data-Pipelines-with-Airflow are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Creating Data Pipelines with Apache Airflow to manage ETL from Amazon S3 into Amazon Redshift☆14Jun 12, 2019Updated 7 years ago
- ☆17Nov 20, 2020Updated 5 years ago
- Code for Data Pipelines with Apache Airflow☆831Aug 15, 2024Updated 2 years ago
- Apache Airflow advanced functionalities examples☆21Mar 22, 2024Updated 2 years ago
- A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Doc…☆25Nov 19, 2024Updated last year
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Final Project for Data Engineering Zoomcamp Course 2024 🧙🔥☆11Apr 17, 2024Updated 2 years ago
- Skeleton project for Apache Airflow training participants to work on.☆17Apr 13, 2026Updated 6 months ago
- ☆21Jan 13, 2024Updated 2 years ago
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆355Jan 12, 2022Updated 4 years ago
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Nov 22, 2021Updated 4 years ago
- Example end to end data engineering project.☆1,448Dec 8, 2022Updated 3 years ago
- Beginner data engineering project - batch edition☆615Apr 13, 2026Updated 6 months ago
- Airflow Examples: code samples for Medium articles☆14Jan 10, 2021Updated 5 years ago
- reating a modern data pipeline using a combination of Terraform, AWS Lambda and S3, Snowflake, DBT, Mage AI, and Dash.☆15Jun 26, 2023Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆167Jun 16, 2020Updated 6 years ago
- A docker based demo to show how we can secure a python webapp and mysql database with Vault☆13Jul 23, 2020Updated 6 years ago
- For this project I am creating an ETL (Extract, Transform, and Load) pipeline using Python, RegEx, and SQL Database. The goal is to retri…☆25Feb 9, 2021Updated 5 years ago
- Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard. The dashboa…☆270Jan 1, 2023Updated 3 years ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆122Jan 8, 2026Updated 9 months ago
- Sample Airflow DAGs☆70Nov 22, 2022Updated 3 years ago
- Big Data Engineering practice project, including ETL with Airflow and Spark using AWS S3 and EMR☆91Jul 17, 2019Updated 7 years ago
- A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract d…☆24Nov 22, 2021Updated 4 years ago
- Python script to import secrets from a KeePass 2X database to hashicorp vault☆16Jan 31, 2026Updated 9 months ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- Example of an ETL Pipeline using Airflow☆39Aug 30, 2017Updated 9 years ago
- Desarrollé un proyecto de ETL sobre archivos de diferentes orígenes (CSV, JSON). Luego, utilicé FastAPI para crear una API que permita re…☆10Dec 9, 2022Updated 3 years ago
- This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and …☆34Feb 9, 2024Updated 2 years ago
- Solution to Data at ANZ virtual internship on Forage☆10May 30, 2021Updated 5 years ago
- End-to-end data engineer project☆25Aug 17, 2023Updated 3 years ago
- Personal Data Engineering Projects☆1,046Feb 8, 2023Updated 3 years ago
- Example DAGs for Airflow 2.9☆13May 20, 2024Updated 2 years ago
- This data project can be used as a take-home assignment to learn Pyspark and Data Engineering.☆20Feb 19, 2023Updated 3 years ago
- Django Project with Gitlab CI☆11Mar 17, 2017Updated 9 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Real-time fraud transaction detection system☆26Aug 28, 2024Updated 2 years ago
- Apache Airflow tutorial☆978Nov 3, 2022Updated 4 years ago
- ☆31Jul 7, 2023Updated 3 years ago
- This project focuses on time series forecasting to predict store sales for Corporation Favorita, a large Ecuadorian-based grocery retaile…☆20Dec 4, 2023Updated 2 years ago
- Data Engineering with AWS, Published by Packt☆344Apr 22, 2026Updated 6 months ago
- A small repository of projects built in my course, REST APIs with Flask and Python.☆21Dec 19, 2017Updated 8 years ago
- This repo is for the Linkedin Learning course: End-to-End Data Engineering Project☆43Nov 9, 2023Updated 2 years ago