Big Data Engineering practice project, including ETL with Airflow and Spark using AWS S3 and EMR
☆91Jul 17, 2019Updated 7 years ago
Alternatives and similar repositories for big-data-engineering-project
Users that are interested in big-data-engineering-project are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- A collection of data engineering projects: data modeling, ETL pipelines, data lakes, infrastructure configuration on AWS, data warehousin…☆15Apr 29, 2021Updated 5 years ago
- My solutions for the Udacity Data Engineering Nanodegree☆33Oct 14, 2019Updated 7 years ago
- With everything I learned from DEZoomcamp from datatalks.club, this project performs a batch processing on AWS for the cycling dataset wh…☆15Jan 4, 2026Updated 9 months ago
- Udacity Data Engineering Nanodegree Project 3☆11Jul 14, 2019Updated 7 years ago
- Predict churn with Apache Spark☆12Feb 2, 2019Updated 7 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Data Engineering, Data Warehouse, Data Mart, Cloud Data, AWS, SAS, Redshift, S3☆36Feb 2, 2021Updated 5 years ago
- Example end to end data engineering project.☆1,448Dec 8, 2022Updated 3 years ago
- Data models, build data warehouses and data lakes, automate data pipelines, and worked with massive datasets.☆12Jul 16, 2019Updated 7 years ago
- Airflow ETL for Meetup API☆44Dec 27, 2018Updated 7 years ago
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆354Jan 12, 2022Updated 4 years ago
- ecommerce GCP Streaming pipeline ― Cloud Storage, Compute Engine, Pub/Sub, Dataflow, Apache Beam, BigQuery and Tableau; GCP Batch pipelin…☆10Mar 9, 2022Updated 4 years ago
- Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake developme…☆2,024Aug 26, 2022Updated 4 years ago
- Data Quest - Data Engineer Learning and Projects☆23May 29, 2019Updated 7 years ago
- Processing TfL data for bike usage with Google Cloud Platform.☆46Jul 15, 2022Updated 4 years ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract d…☆24Nov 22, 2021Updated 4 years ago
- This repo contains commands that data engineers use in day to day work.☆64Feb 4, 2023Updated 3 years ago
- Projects done in the Data Engineering Nanodegree by Udacity.com☆271Mar 1, 2026Updated 8 months ago
- Beginner data engineering project - batch edition☆608Apr 13, 2026Updated 6 months ago
- A real-time event pipeline around Kafka Ecosystem for Chicago Transit Authority.☆32Aug 14, 2023Updated 3 years ago
- Infuse AI into your application. Create and deploy a customer churn prediction model with IBM Cloud Private for Data, Db2 Warehouse, Spar…☆18Sep 17, 2025Updated last year
- Simple ETL pipeline using Python☆29May 22, 2023Updated 3 years ago
- ☆26Feb 2, 2018Updated 8 years ago
- Data pipeline project using Data Factory, Databricks and Cosmosdb Graph, deployed using Azure DevOps, secured using firewalls and Azure A…☆11Dec 14, 2022Updated 3 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Resources and projects from Udacity Data Engineering with AWS nano degree programme☆30Apr 12, 2023Updated 3 years ago
- More than 2000+ Data engineer interview questions.☆1,807Jan 13, 2026Updated 9 months ago
- This GitHub project provides a series of lab exercises which help users get started using the Redshift platform.☆53Mar 31, 2021Updated 5 years ago
- This repo consists of all important concepts for data engineers.☆11Jun 2, 2026Updated 5 months ago
- An end-to-end data engineering pipeline to create a dashboard for the latest content on the r/Stocks subreddit☆20Aug 5, 2022Updated 4 years ago
- ( These solutions tested on 4 node Hortonwork cluster on my laptop. Do not test on your production environment until you test... :)☆20Apr 18, 2020Updated 6 years ago
- A terraform project to install as AWS hosted wordpress site☆12Dec 31, 2016Updated 9 years ago
- ETL (Extract, Transform and Load) with the Spark Python API (PySpark) and Hadoop Distributed File System (HDFS)☆17Dec 18, 2018Updated 7 years ago
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Aug 14, 2023Updated 3 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- A Covid-19 data pipeline on AWS featuring PySpark/Glue, Docker, Great Expectations, Airflow, and Redshift, templated in CloudFormation an…☆24Nov 21, 2023Updated 2 years ago
- Udacity Data Engineering Nano Degree (DEND)☆187Jan 20, 2020Updated 6 years ago
- This project walks through integrating JFrog Artifactory with AWS CodePipeline☆12Mar 6, 2025Updated last year
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Nov 22, 2021Updated 4 years ago
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆167Jun 16, 2020Updated 6 years ago
- ☆218Aug 13, 2023Updated 3 years ago
- This workshop is meant to give customers a hands-on experience with mentioned AWS services. Serverless Data Lake workshop helps customer…☆38Feb 9, 2021Updated 5 years ago