This is a capstone project that entails building an end-to-end ETL (Extract-Transform-Load) Data pipeline which extracts UK accident and traffic datasets from Amazon S3, clean and transform with Pyspark, transfer it back to S3 and finally load to Amazon Redshift (Distributed Database), from where the data can be queried for ad-hoc analyses.
☆18Jun 6, 2020Updated 6 years ago
Alternatives and similar repositories for UK_Accident_Traffic_ETL_Pipeline
Users that are interested in UK_Accident_Traffic_ETL_Pipeline are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- implementing an end-to-end tweets ETL/Analysis pipeline.☆59Dec 8, 2022Updated 3 years ago
- ETL using Python in Jupyter Notebook, loading CSV, cleaning data, and saving to SQL Database.☆14Nov 17, 2020Updated 5 years ago
- Primer curso de Craftech Academy - Marzo 2021☆12Aug 3, 2021Updated 5 years ago
- ☆15Jan 16, 2024Updated 2 years ago
- Teaching notes from my Advanced SQL workshops as local lead instructor at General Assembly New York. The first edition was created for th…☆19Feb 14, 2020Updated 6 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- Gojek UI slicing built with Flutter for a semester 3 final project.☆11Dec 10, 2024Updated last year
- A clean and scalable backend system for payment processing, designed with microservices architecture. It includes user management, order …☆11Jan 19, 2025Updated last year
- Create a data pipeline on AWS to execute batch processing in a Spark cluster provisioned by Amazon EMR. ETL using managed airflow: extrac…☆10Jul 12, 2021Updated 5 years ago
- A repo to track data engineering projects☆14Nov 11, 2022Updated 3 years ago
- Build Your Own Roadmap☆11Jul 8, 2020Updated 6 years ago
- Latex Template for Thesis at DTETI FT UGM☆20May 12, 2026Updated 3 months ago
- 🏟☆28Nov 11, 2020Updated 5 years ago
- A collection of data engineering projects: data modeling, ETL pipelines, data lakes, infrastructure configuration on AWS, data warehousin…☆15Apr 29, 2021Updated 5 years ago
- Guide to CS Engineering and Interview Prep☆18Dec 26, 2024Updated last year
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆17Oct 1, 2019Updated 6 years ago
- An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.☆1,539Mar 9, 2020Updated 6 years ago
- Loan Default Prediction using PySpark, with jobs scheduled by Apache Airflow and Integration with Spark using Apache Livy☆22Dec 26, 2020Updated 5 years ago
- Leetcode solution for weekly contest☆16Jan 11, 2020Updated 6 years ago
- Samples of ML models learning from source code☆20Nov 28, 2022Updated 3 years ago
- A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract d…☆24Nov 22, 2021Updated 4 years ago
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Aug 14, 2023Updated 3 years ago
- A real-time event pipeline around Kafka Ecosystem for Chicago Transit Authority.☆33Aug 14, 2023Updated 3 years ago
- Steven's 100DaysOfCloudRepo☆17Nov 22, 2020Updated 5 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Enriching Your Python Classes With Dunder (Magic, Special) Methods☆20Jun 26, 2017Updated 9 years ago
- A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…☆29Aug 8, 2020Updated 6 years ago
- These are the Jupyter notebooks for the Big Data specialization in the Data Science Program.☆15Apr 3, 2020Updated 6 years ago
- My Data Engineer Capstone project. A consolidated dataset with several jobs around the world.☆13May 22, 2023Updated 3 years ago
- Deployable and containerized CNN for detection of malaria parasites in blood smears where images have been segmented into single cells.☆58Jun 26, 2023Updated 3 years ago
- Project files for the post: Running PySpark Applications on Amazon EMR using Apache Airflow: Using the new Amazon Managed Workflows for A…☆41Jul 6, 2022Updated 4 years ago
- in this prj we will find and detect the lane and we are able to find the area that the car should be in☆24Mar 28, 2022Updated 4 years ago
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆352Jan 12, 2022Updated 4 years ago
- This is where we put useful code for our daily job with data.☆28Mar 19, 2025Updated last year
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- This repository contains everything you need to become proficient in System Design and Case Studies with Code Implementation☆21Jan 27, 2024Updated 2 years ago
- My Git Repo for Csv Data☆21Oct 5, 2025Updated 10 months ago
- A simple script to fetch files from a google drive folder and upload to S3☆23Sep 15, 2016Updated 9 years ago
- Query Optimizer Service☆55Aug 1, 2026Updated 2 weeks ago
- SQL data model for working with Snowplow web data. Supports Redshift and Looker. Snowflake and BigQuery coming soon☆60Dec 1, 2020Updated 5 years ago
- ☆24Dec 20, 2022Updated 3 years ago
- Udacity Data Engineering Nanodegree Program☆54Mar 4, 2021Updated 5 years ago