This is a capstone project that entails building an end-to-end ETL (Extract-Transform-Load) Data pipeline which extracts UK accident and traffic datasets from Amazon S3, clean and transform with Pyspark, transfer it back to S3 and finally load to Amazon Redshift (Distributed Database), from where the data can be queried for ad-hoc analyses.
☆18Jun 6, 2020Updated 6 years ago
Alternatives and similar repositories for UK_Accident_Traffic_ETL_Pipeline
Users that are interested in UK_Accident_Traffic_ETL_Pipeline are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- implementing an end-to-end tweets ETL/Analysis pipeline.☆59Dec 8, 2022Updated 3 years ago
- Data Science, Visualization, and Predictive Analytics: Kaggle Dataset - 1.6M accidents & traffic flow over 16 years☆17Feb 10, 2018Updated 8 years ago
- ETL using Python in Jupyter Notebook, loading CSV, cleaning data, and saving to SQL Database.☆14Nov 17, 2020Updated 5 years ago
- Spark + Python for Maketing Analytics☆10Apr 19, 2017Updated 9 years ago
- ☆12Sep 19, 2021Updated 5 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Teaching notes from my Advanced SQL workshops as local lead instructor at General Assembly New York. The first edition was created for th…☆19Feb 14, 2020Updated 6 years ago
- Example project for consuming AWS Kinesis streamming and save data on Amazon Redshift using Apache Spark☆11May 22, 2018Updated 8 years ago
- Primer curso de Craftech Academy - Marzo 2021☆12Aug 3, 2021Updated 5 years ago
- ☆10May 18, 2020Updated 6 years ago
- For this project I am creating an ETL (Extract, Transform, and Load) pipeline using Python, RegEx, and SQL Database. The goal is to retri…☆25Feb 9, 2021Updated 5 years ago
- A repo to track data engineering projects☆14Nov 11, 2022Updated 3 years ago
- CICD pipeline that deploys a dbt image on a GKE cluster☆11Jul 7, 2021Updated 5 years ago
- This is a collection of computational illustrations, in MATLAB and in Python, accompanying the book "Mathematical Pictures at a Data Scie…☆12Aug 26, 2022Updated 4 years ago
- Data mining algorithms with Python☆10Jun 26, 2019Updated 7 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- An SQL analysis of traffic accidents within the City of Chicago limits and under the jurisdiction of Chicago Police Department (CPD).☆21Feb 11, 2023Updated 3 years ago
- Insight Data Engineering project: A platform built in HDFS, Spark and Airflow to help you to find social influencers from GitHub Net…☆16May 21, 2024Updated 2 years ago
- A collection of data engineering projects: data modeling, ETL pipelines, data lakes, infrastructure configuration on AWS, data warehousin…☆15Apr 29, 2021Updated 5 years ago
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆17Oct 1, 2019Updated 7 years ago
- ☆12Jul 13, 2022Updated 4 years ago
- Tweepy Stream Example☆19Apr 23, 2019Updated 7 years ago
- Self-study plan to achieve mastery in data science☆30Mar 29, 2023Updated 3 years ago
- ☆16Mar 5, 2025Updated last year
- Infrastructure for researching self-driving databases☆34Jul 2, 2025Updated last year
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Leetcode solution for weekly contest☆16Jan 11, 2020Updated 6 years ago
- Collection of notebooks☆17Oct 27, 2024Updated 2 years ago
- Apache Spark 2 for Beginners, published by Packt☆33Oct 31, 2022Updated 4 years ago
- ☆14Aug 9, 2016Updated 10 years ago
- Exploring the use of options in creating small worlds for faster learning in RL Domains☆16Jan 23, 2012Updated 14 years ago
- A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract d…☆24Nov 22, 2021Updated 4 years ago
- Samples of ML models learning from source code☆21Nov 28, 2022Updated 3 years ago
- A complete example of an AWS Glue application that uses the Serverless Framework to deploy the infrastructure and DevContainers and/or Do…☆22Updated this week
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Aug 14, 2023Updated 3 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- Steven's 100DaysOfCloudRepo☆17Nov 22, 2020Updated 5 years ago
- An open-source repo to product management case studies.☆34Updated this week
- My Data Engineer Capstone project. A consolidated dataset with several jobs around the world.☆13May 22, 2023Updated 3 years ago
- Mastering Spark for Data Science, published by Packt☆51Apr 22, 2026Updated 6 months ago
- A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…☆29Aug 8, 2020Updated 6 years ago
- All Coding project for CS6515 GA☆17Jul 22, 2022Updated 4 years ago
- Enriching Your Python Classes With Dunder (Magic, Special) Methods☆20Jun 26, 2017Updated 9 years ago