This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.
☆24Aug 21, 2025Updated last year
Alternatives and similar repositories for Python-ETL-pipeline-using-Airflow-on-AWS
Users that are interested in Python-ETL-pipeline-using-Airflow-on-AWS are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Integrating with Spotify API and extracting Data. Deploying code on AWS Lambda for Data Extraction. Adding trigger to run the extraction …☆12Jul 5, 2023Updated 3 years ago
- Integrating Apache Airflow, dbt, Great Expectations and Apache Superset to develop a modern open source data stack.☆19Jun 19, 2022Updated 4 years ago
- My solutions for the Udacity Data Engineering Nanodegree☆33Oct 14, 2019Updated 7 years ago
- Udacity Data Engineering Nano Degree Project, Data Modeling for fact and dimension tables, and ETL pipeline that transfers data from file…☆10Dec 12, 2020Updated 5 years ago
- Local SQL Database ---> Azure ---> Power BI☆15Oct 13, 2023Updated 3 years ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- A collection of FIO test cases and a Python script to run them.☆11Sep 5, 2023Updated 3 years ago
- ☆30Jun 12, 2023Updated 3 years ago
- Spotify ETL Pipeline☆13Oct 13, 2023Updated 3 years ago
- Simple project using pyflink, kafka and postgre containerized using Docker☆12Aug 26, 2024Updated 2 years ago
- ☆15Sep 14, 2021Updated 5 years ago
- This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. Ther…☆28Aug 22, 2025Updated last year
- Examples of Clean Architecture in Python, with folders matching the classic circle diagram.☆13Oct 9, 2026Updated 2 weeks ago
- An AWS Data Engineering End-to-End Project (Glue, Lambda, Kinesis, Redshift, QuickSight, Athena, EC2, S3)☆17Sep 20, 2023Updated 3 years ago
- This repo includes all exercises for courses and projects that I have finished on datacamp.☆23Sep 12, 2026Updated last month
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- ☆16Feb 20, 2026Updated 8 months ago
- Low-code/No-code development platform☆12Dec 26, 2023Updated 2 years ago
- ☆11Oct 8, 2021Updated 5 years ago
- Toy bitemporal key-value database and visualizer☆12Jun 6, 2026Updated 4 months ago
- Built a Data Pipeline for a Retail store using AWS services that collects data from its transactional database (OLTP) in Snowflake and tr…☆14May 25, 2023Updated 3 years ago
- Training of Noise-to-Image Diffusion Model on Multi-Channel Brain Tumor MRI Scans.☆12Jan 12, 2023Updated 3 years ago
- Udacity data engineering nanodegree project☆23Nov 8, 2019Updated 6 years ago
- This repository contains code and configuration files for an Extract, Transform, Load (ETL) project using Google Cloud Data Fusion for da…☆21Feb 23, 2024Updated 2 years ago
- This is a demo repository for parallel multi-index question answering using streamlit and llama index☆23Aug 31, 2023Updated 3 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- ☆23Oct 21, 2024Updated 2 years ago
- Tool which summarizes daily and total gas consumption of all transactions sent from a specified Ethereum address.☆15Jun 28, 2023Updated 3 years ago
- 📈 A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kuberne…☆143Sep 27, 2026Updated last month
- Benchmarking RAG and agentic systems in Go vs Python☆12Oct 2, 2025Updated last year
- Learn Retrieval-Augmented Generation (RAG) from Scratch using LLMs from Hugging Face and Langchain or Python☆13Jan 20, 2025Updated last year
- This repo contains my projects from the Udacity Data Engineering Nano degree☆14Apr 26, 2023Updated 3 years ago
- ImageTester is a Cli tool to perform visual tests on images, PDF files, Office files, and more!☆11Updated this week
- ☆17Oct 4, 2025Updated last year
- Curated list of AI tools for sales automation, LinkedIn outreach, cold email, and B2B lead generation.☆22Jun 21, 2025Updated last year
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- # **ABSTRACT** Main Objective: The main agenda of this project is: Perform extensive Exploratory Data Analys…☆33Aug 30, 2021Updated 5 years ago
- This project aims to leverage Amazon Web Services to create trending Youtube videos analytics service. Project contains different data en…☆23Aug 16, 2025Updated last year
- This is a project of mine which includes all the python skills I have learnt ! This is a virtual desktop as well as web assistant ! The m…☆12Nov 3, 2020Updated 5 years ago
- Classwork projects and home works done through Udacity data engineering nano degree☆10Jun 6, 2021Updated 5 years ago
- Python projects for beginners, intermediate and advanced devs☆27Jul 24, 2026Updated 3 months ago
- Wanted to integrate Julius AI in my workflows but couldn't hence this.☆15Jan 1, 2025Updated last year
- esbuild plugin for rendering markdown as React components☆13Feb 2, 2021Updated 5 years ago