A collection of data engineering projects: data modeling, ETL pipelines, data lakes, infrastructure configuration on AWS, data warehousing, containerization, and a dashboard to monitor data pipeline KPIs
☆15Apr 29, 2021Updated 5 years ago
Alternatives and similar repositories for Data_Engineering_Projects
Users that are interested in Data_Engineering_Projects are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆16May 14, 2024Updated 2 years ago
- Set up a Cost-Effective Modern Data Stack for a Charity☆19Mar 26, 2025Updated last year
- capstone project for Dataengineer.io bootcamp Public Repo☆12Feb 20, 2024Updated 2 years ago
- Hands-On Low-Code Application Development with Salesforce, published by Packt☆11Jan 18, 2023Updated 3 years ago
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆17Oct 1, 2019Updated 6 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- Modern Data Stack in a box with dbt-duckdb and Apache Superset☆16Updated this week
- CI/CD repository template to automate deployments of your production flows☆15Jul 1, 2024Updated 2 years ago
- This repo consists of all important concepts for data engineers.☆11Jun 2, 2026Updated last month
- 🎩 AI-powered cover letter generator☆27Jul 13, 2025Updated last year
- Example project for consuming AWS Kinesis streamming and save data on Amazon Redshift using Apache Spark☆11May 22, 2018Updated 8 years ago
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Aug 14, 2023Updated 2 years ago
- Scan and monitor your network effortlessly! Nmap Prometheus Exporter provides insights into network health and security with Prometheus-c…☆15Oct 2, 2023Updated 2 years ago
- 😈Complete End to End ETL Pipeline with Spark, Airflow, & AWS☆51Aug 23, 2019Updated 6 years ago
- Udacity Data Engineering Nanodegree Project 3☆12Jul 14, 2019Updated 7 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validatio…☆56May 6, 2023Updated 3 years ago
- Create a data pipeline on AWS to execute batch processing in a Spark cluster provisioned by Amazon EMR. ETL using managed airflow: extrac…☆10Jul 12, 2021Updated 5 years ago
- Course Material Data Engineering on AWS Course☆31Sep 9, 2024Updated last year
- Match previews for 1770 games in the English Premier League☆10Aug 27, 2020Updated 5 years ago
- An e-commerce backend made with Go (WIP)☆11Sep 1, 2023Updated 2 years ago
- A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Doc…☆25Nov 19, 2024Updated last year
- ☆16Feb 20, 2026Updated 5 months ago
- Data Engineering Hours With Experts Coding Challenge☆13Mar 16, 2026Updated 4 months ago
- This data project can be used as a take-home assignment to learn Pyspark and Data Engineering.☆20Feb 19, 2023Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ☆17Mar 26, 2023Updated 3 years ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆24May 14, 2022Updated 4 years ago
- Code for the Data Engineering Zoomcamp☆20Dec 12, 2022Updated 3 years ago
- ☆36Feb 6, 2024Updated 2 years ago
- A Covid-19 data pipeline on AWS featuring PySpark/Glue, Docker, Great Expectations, Airflow, and Redshift, templated in CloudFormation an…☆24Nov 21, 2023Updated 2 years ago
- scaleway deploy container with registry and custom endpoint dns☆11Jul 24, 2024Updated last year
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆48Dec 11, 2023Updated 2 years ago
- This project aims to build a traveling recommendation application using Google Places API and OpenAI LLM.☆11Mar 19, 2024Updated 2 years ago
- SQLMesh example projects☆42Jul 2, 2025Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- A multi-community spatio-temporal graph convolutional network (MC_STGCN) for passenger demand forecasting at multi-region level☆10Sep 28, 2020Updated 5 years ago
- Whatsapp Automation☆15May 6, 2024Updated 2 years ago
- Simple ETL pipeline using Python☆29May 22, 2023Updated 3 years ago
- data science interview questions company wise which include the data analyst , junior data scientist , machine learning engineer etc. pos…☆17Apr 20, 2022Updated 4 years ago
- Case study solutions for #8WeekSQLChallenge by Danny Ma.☆44Jan 9, 2023Updated 3 years ago
- 💯 Materials to help you rock your next coding interview☆12Aug 18, 2019Updated 6 years ago
- Tweepy Stream Example☆19Apr 23, 2019Updated 7 years ago