This is a simple ETL using Airflow. First, we fetch data from API (extract). Then, we drop unused columns, convert to CSV, and validate (transform). Finally, we load the transformed data to database (load).
☆24Oct 12, 2019Updated 6 years ago
Alternatives and similar repositories for airflow-etl-learn
Users that are interested in airflow-etl-learn are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Udacity Data Engineering Nano Degree Project, Data Modeling for fact and dimension tables, and ETL pipeline that transfers data from file…☆10Dec 12, 2020Updated 5 years ago
- Create a data pipeline on AWS to execute batch processing in a Spark cluster provisioned by Amazon EMR. ETL using managed airflow: extrac…☆10Jul 12, 2021Updated 5 years ago
- An ETL pipeline that extracts data from S3, stages them in Redshift, and transforms data into a set of dimensional tables☆16May 5, 2020Updated 6 years ago
- Training two models, one with with AutoML & one with HyperDrive, compare, and deploy the best model as a service - A Machine Learning Eng…☆16Jan 25, 2022Updated 4 years ago
- Local SQL Database ---> Azure ---> Power BI☆15Oct 13, 2023Updated 2 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- A guide to show you how to import data for ETL☆21Dec 8, 2022Updated 3 years ago
- Data Vault 2.0: Code generation, Vertica, Airflow☆12Nov 20, 2019Updated 6 years ago
- Handle Big Data for Machine Learning using Python and PySpark, Building ETL Pipelines with PySpark, MongoDB, and Bokeh☆10Nov 12, 2021Updated 4 years ago
- ☆10Oct 20, 2022Updated 3 years ago
- An Airflow pipeline for the collection of historical Twitter data☆10Aug 5, 2019Updated 7 years ago
- Matching messy Pandas columns with FuzzyWuzzy (Medium Article)☆13Sep 29, 2019Updated 6 years ago
- Template to deploy Synapse Analytics using best practices to deliver a proof of concept.☆21Mar 3, 2023Updated 3 years ago
- This repo contains implementation of various functionalities of various message queues in Python.☆13Aug 13, 2020Updated 6 years ago
- Udacity data engineering nanodegree project☆22Nov 8, 2019Updated 6 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- UNIX top-like script for monitoring SASWORK and SASUTIL directories. If you find this useful then check out ESM for SAS® -☆17Feb 28, 2020Updated 6 years ago
- Use fastAPI to generate html web app that will serve a local directory or S3 bucket of images☆11Jan 18, 2021Updated 5 years ago
- ⌗ Learn PostgreSql.☆19Oct 28, 2025Updated 10 months ago
- ☆27Mar 7, 2022Updated 4 years ago
- Data Visualizations for New York City☆11Apr 15, 2020Updated 6 years ago
- ☆31Jun 12, 2023Updated 3 years ago
- Active Statistics book web page☆12Jan 3, 2025Updated last year
- Spark data pipeline that processes movie ratings data.☆31Aug 1, 2026Updated last month
- A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…☆29Aug 8, 2020Updated 6 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- A simple Python Twitter Reply Bot which was made using Tweepy☆17Dec 12, 2022Updated 3 years ago
- This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. Ther…☆29Aug 22, 2025Updated last year
- Blockchain ETL Architecture☆47Oct 10, 2022Updated 3 years ago
- ☆14May 12, 2026Updated 3 months ago
- An introduction to the Observer design patten with C++ using a Weather Station Example☆20Apr 10, 2019Updated 7 years ago
- MLflow logging for PyMC☆14Aug 14, 2026Updated 3 weeks ago
- ☆12Nov 6, 2024Updated last year
- give me a toml configuration file, I'll give you export MY_ENV=foo☆10Jul 6, 2018Updated 8 years ago
- A Spotify Playlist Recommendation System based on a collaborative filtering algorithm using the Million Playlist Data Set (MPD).☆17Dec 8, 2022Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Slides and materials for workshop on "Two views on regression with PyMC3 and scikit-learn"☆18Apr 6, 2023Updated 3 years ago
- Timbre Transfer using Differentiable Digital Signal Processing based on DDSP repository☆42Apr 24, 2020Updated 6 years ago
- Python implementation of plot from Kay, Kola, Hullman, Munson "When (ish) is My Bus?" (2016)☆18Dec 19, 2019Updated 6 years ago
- Mastering Gaussian Processes with PyMC☆17Aug 18, 2025Updated last year
- FastAPI Template☆11Jan 11, 2025Updated last year
- C++ programs to transfer a Text file from server to client using TCP☆24Jan 30, 2025Updated last year
- Advent of Code 2025 done in Lean 4☆18Dec 12, 2025Updated 8 months ago