Create a data pipeline on AWS to execute batch processing in a Spark cluster provisioned by Amazon EMR. ETL using managed airflow: extracts data from S3, transform data using spark, load transformed data back to S3.
☆10Jul 12, 2021Updated 5 years ago
Alternatives and similar repositories for Batch-ETL-with-AWS-EMR-and-MWAA
Users that are interested in Batch-ETL-with-AWS-EMR-and-MWAA are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆11Dec 4, 2023Updated 2 years ago
- This project aims to rate football players using data and statistics recorded from the last match they participated in. Much of the code …☆13Nov 22, 2021Updated 4 years ago
- ☆11Sep 30, 2022Updated 4 years ago
- A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…☆29Aug 8, 2020Updated 6 years ago
- Spark data pipeline that processes movie ratings data.☆31Updated this week
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ☆22Jul 29, 2024Updated 2 years ago
- Solution to Data at ANZ virtual internship on Forage☆10May 30, 2021Updated 5 years ago
- Example project for consuming AWS Kinesis streamming and save data on Amazon Redshift using Apache Spark☆12May 22, 2018Updated 8 years ago
- ☆36Feb 19, 2026Updated 8 months ago
- Repository created to host udacity data engineer exercises☆11Mar 1, 2026Updated 8 months ago
- Image Segmentation using Fully Convolutional Networks in PyTorch☆11May 16, 2019Updated 7 years ago
- An interactive repository of AWS Certified Cloud Practitioner questions and answers, designed for effective exam preparation.☆13Oct 20, 2024Updated 2 years ago
- Alzheimer’s Disease (AD) is a neurological brain disorder marked by dementia and neurological dysfunction that affects memory, behavioral…☆16Aug 28, 2022Updated 4 years ago
- This project focuses on building a robust data pipeline using Apache Airflow to automate the ingestion of weather data from the OpenWeath…☆22Feb 3, 2026Updated 9 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ☆21Jan 13, 2024Updated 2 years ago
- Online Color Palette (https://youtu.be/ig91zc-ERSE)☆16Jun 24, 2022Updated 4 years ago
- 🚚 ETL for Spark and Airflow☆25Mar 19, 2018Updated 8 years ago
- Alzheimer's / dementia progression classifier for MRIs using CNNs and transfer learning☆19Jan 22, 2018Updated 8 years ago
- Delta Live Tables Workshop Resources☆17Feb 24, 2023Updated 3 years ago
- A repo to track data engineering projects☆14Nov 11, 2022Updated 3 years ago
- Power Pop Health is a collection of content intended to simplify the process of ingesting and prepping Healthcare Open Data using Azure d…☆18May 23, 2022Updated 4 years ago
- Insight Data Engineering project: A platform built in HDFS, Spark and Airflow to help you to find social influencers from GitHub Net…☆16May 21, 2024Updated 2 years ago
- This is a simple ETL using Airflow. First, we fetch data from API (extract). Then, we drop unused columns, convert to CSV, and validate (…☆24Oct 12, 2019Updated 7 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- A collection of data engineering projects: data modeling, ETL pipelines, data lakes, infrastructure configuration on AWS, data warehousin…☆15Apr 29, 2021Updated 5 years ago
- ☆27Mar 7, 2022Updated 4 years ago
- Tutorials of data science concepts and packages in Python☆21Feb 11, 2016Updated 10 years ago
- Databricks CI/CD using Azure DevOps☆21Nov 1, 2022Updated 4 years ago
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆17Oct 1, 2019Updated 7 years ago
- CodeSignal CodeFights SQL Database queries☆23Dec 26, 2019Updated 6 years ago
- Repositorio con info basica de archivos y documentos para la visualización de datos en fútbol y Python☆79May 26, 2025Updated last year
- PySpark functions and utilities with examples. Assists ETL process of data modeling☆104Dec 3, 2020Updated 5 years ago
- ☆10Sep 10, 2022Updated 4 years ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- Tweepy Stream Example☆19Apr 23, 2019Updated 7 years ago
- GUI Dataset Collector: A Tool for Capturing and Annotating GUI Interactions with annotations in COCO format☆13Aug 18, 2025Updated last year
- Image Classification with transfer learning | a PyTorch Tutorial to Transfer Learning☆21Jul 25, 2024Updated 2 years ago
- A webscraper that captures search results data from www.amazon.com☆25Dec 21, 2020Updated 5 years ago
- Infrastructure for researching self-driving databases☆34Jul 2, 2025Updated last year
- This is a capstone project that entails building an end-to-end ETL (Extract-Transform-Load) Data pipeline which extracts UK accident and …☆18Jun 6, 2020Updated 6 years ago
- In this project I used apache airflow to scrape website periodically. This is for the tutorials I do on youtube.☆10Nov 21, 2022Updated 3 years ago