AdeboyeML / UK_Accident_Traffic_ETL_PipelineView external linksLinks
This is a capstone project that entails building an end-to-end ETL (Extract-Transform-Load) Data pipeline which extracts UK accident and traffic datasets from Amazon S3, clean and transform with Pyspark, transfer it back to S3 and finally load to Amazon Redshift (Distributed Database), from where the data can be queried for ad-hoc analyses.
☆18Jun 6, 2020Updated 5 years ago
Alternatives and similar repositories for UK_Accident_Traffic_ETL_Pipeline
Users that are interested in UK_Accident_Traffic_ETL_Pipeline are comparing it to the libraries listed below
Sorting:
- implementing an end-to-end tweets ETL/Analysis pipeline.☆59Dec 8, 2022Updated 3 years ago
- Primer curso de Craftech Academy - Marzo 2021☆11Aug 3, 2021Updated 4 years ago
- A summary of useful resources in order to learn about AI☆10May 9, 2020Updated 5 years ago
- Data mining algorithms with Python☆10Jun 26, 2019Updated 6 years ago
- The gaming industry is certainly one of the thriving industries of the modern age and one of those that are most influenced by the advanc…☆12Jun 29, 2020Updated 5 years ago
- CICD pipeline that deploys a dbt image on a GKE cluster☆11Jul 7, 2021Updated 4 years ago
- Example project for consuming AWS Kinesis streamming and save data on Amazon Redshift using Apache Spark☆11May 22, 2018Updated 7 years ago
- All Coding project for CS6515 GA☆14Jul 22, 2022Updated 3 years ago
- Build Your Own Roadmap☆11Jul 8, 2020Updated 5 years ago
- ☆15Dec 2, 2020Updated 5 years ago
- Teaching notes from my Advanced SQL workshops as local lead instructor at General Assembly New York. The first edition was created for th…☆18Feb 14, 2020Updated 6 years ago
- Samples of ML models learning from source code☆20Nov 28, 2022Updated 3 years ago
- data visualizations and R code for #TidyTuesday 2021☆16Feb 4, 2022Updated 4 years ago
- A repo to track data engineering projects☆13Nov 11, 2022Updated 3 years ago
- Leetcode solution for weekly contest☆16Jan 11, 2020Updated 6 years ago
- These are the Jupyter notebooks for the Big Data specialization in the Data Science Program.☆15Apr 3, 2020Updated 5 years ago
- Insight Data Engineering project: A platform built in HDFS, Spark and Airflow to help you to find social influencers from GitHub Net…☆16May 21, 2024Updated last year
- This Challenge aims to infer important COVID-19 public health risk factors from outdated data in South Africa☆20Dec 8, 2022Updated 3 years ago
- Collection of notebooks☆17Oct 27, 2024Updated last year
- A collection of data engineering projects: data modeling, ETL pipelines, data lakes, infrastructure configuration on AWS, data warehousin…☆15Apr 29, 2021Updated 4 years ago
- This repository explains how to predict customer churn. An Hackathon Organized by Data Science Nigeria(DSN-AI) to help Expresso predict c…☆21Oct 17, 2021Updated 4 years ago
- My Data Engineer Capstone project. A consolidated dataset with several jobs around the world.☆13May 22, 2023Updated 2 years ago
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆17Oct 1, 2019Updated 6 years ago
- My Git Repo for Csv Data☆21Oct 5, 2025Updated 4 months ago
- Loan Default Prediction using PySpark, with jobs scheduled by Apache Airflow and Integration with Spark using Apache Livy☆20Dec 26, 2020Updated 5 years ago
- An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.☆1,485Mar 9, 2020Updated 5 years ago
- Tweepy Stream Example☆19Apr 23, 2019Updated 6 years ago
- A simple spark standalone cluster for your testing environment purposses☆23Jul 25, 2020Updated 5 years ago
- Demonstration of using Apache Spark to build robust ETL pipelines while taking advantage of open source, general purpose cluster computin…☆24Aug 11, 2023Updated 2 years ago
- This is where we put useful code for our daily job with data.☆27Mar 19, 2025Updated 10 months ago
- This repository contains all the python projects done as a tutorial☆25Mar 27, 2025Updated 10 months ago
- #TidyTuesday Contributions.☆31Aug 29, 2022Updated 3 years ago
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Aug 14, 2023Updated 2 years ago
- A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…☆29Aug 8, 2020Updated 5 years ago
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆347Jan 12, 2022Updated 4 years ago
- Project files for the post: Running PySpark Applications on Amazon EMR using Apache Airflow: Using the new Amazon Managed Workflows for A…☆41Jul 6, 2022Updated 3 years ago
- Demo code to illustrate the execution of PyTest unit test cases for AWS Glue jobs in AWS CodePipeline using AWS CodeBuild projects☆49Dec 3, 2024Updated last year
- Ravi Azure ADB ADF Repository☆64Jan 25, 2025Updated last year
- SageMaker specific extensions to TensorFlow.☆54Jul 23, 2024Updated last year