This is a capstone project that entails building an end-to-end ETL (Extract-Transform-Load) Data pipeline which extracts UK accident and traffic datasets from Amazon S3, clean and transform with Pyspark, transfer it back to S3 and finally load to Amazon Redshift (Distributed Database), from where the data can be queried for ad-hoc analyses.
☆18Jun 6, 2020Updated 6 years ago
Alternatives and similar repositories for UK_Accident_Traffic_ETL_Pipeline
Users that are interested in UK_Accident_Traffic_ETL_Pipeline are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- implementing an end-to-end tweets ETL/Analysis pipeline.☆59Dec 8, 2022Updated 3 years ago
- ETL using Python in Jupyter Notebook, loading CSV, cleaning data, and saving to SQL Database.☆14Nov 17, 2020Updated 5 years ago
- Spark + Python for Maketing Analytics☆10Apr 19, 2017Updated 9 years ago
- Primer curso de Craftech Academy - Marzo 2021☆12Aug 3, 2021Updated 5 years ago
- Teaching notes from my Advanced SQL workshops as local lead instructor at General Assembly New York. The first edition was created for th…☆19Feb 14, 2020Updated 6 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- For this project I am creating an ETL (Extract, Transform, and Load) pipeline using Python, RegEx, and SQL Database. The goal is to retri…☆25Feb 9, 2021Updated 5 years ago
- Create a data pipeline on AWS to execute batch processing in a Spark cluster provisioned by Amazon EMR. ETL using managed airflow: extrac…☆10Jul 12, 2021Updated 5 years ago
- A repo to track data engineering projects☆14Nov 11, 2022Updated 3 years ago
- CICD pipeline that deploys a dbt image on a GKE cluster☆11Jul 7, 2021Updated 5 years ago
- Build Your Own Roadmap☆11Jul 8, 2020Updated 6 years ago
- Data mining algorithms with Python☆10Jun 26, 2019Updated 7 years ago
- Roadmap to becoming a web developer in 2017 in spanish, Roadmap para ser un desarrollador web en el 2017☆15Jun 16, 2017Updated 9 years ago
- An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.☆1,547Mar 9, 2020Updated 6 years ago
- Tweepy Stream Example☆19Apr 23, 2019Updated 7 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Loan Default Prediction using PySpark, with jobs scheduled by Apache Airflow and Integration with Spark using Apache Livy☆22Dec 26, 2020Updated 5 years ago
- Samples of ML models learning from source code☆21Nov 28, 2022Updated 3 years ago
- Apache Spark 2 for Beginners, published by Packt☆33Oct 31, 2022Updated 3 years ago
- Infrastructure for researching self-driving databases☆34Jul 2, 2025Updated last year
- data visualizations and R code for #TidyTuesday 2021☆16Feb 4, 2022Updated 4 years ago
- Exploring the use of options in creating small worlds for faster learning in RL Domains☆16Jan 23, 2012Updated 14 years ago
- Package for Computational Biology Reading Group☆13Apr 20, 2022Updated 4 years ago
- All Coding project for CS6515 GA☆17Jul 22, 2022Updated 4 years ago
- A complete example of an AWS Glue application that uses the Serverless Framework to deploy the infrastructure and DevContainers and/or Do…☆22Oct 6, 2026Updated last week
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract d…☆24Nov 22, 2021Updated 4 years ago
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Aug 14, 2023Updated 3 years ago
- Demonstration of using Apache Spark to build robust ETL pipelines while taking advantage of open source, general purpose cluster computin…☆24Aug 11, 2023Updated 3 years ago
- An open-source repo to product management case studies.☆33Updated this week
- A real-time event pipeline around Kafka Ecosystem for Chicago Transit Authority.☆32Aug 14, 2023Updated 3 years ago
- Steven's 100DaysOfCloudRepo☆17Nov 22, 2020Updated 5 years ago
- Enriching Your Python Classes With Dunder (Magic, Special) Methods☆20Jun 26, 2017Updated 9 years ago
- Spark data pipeline that processes movie ratings data.☆31Oct 1, 2026Updated 2 weeks ago
- Apache Spark 3 for Data Engineering and Analytics with Python , By Packt publishing☆24Jul 23, 2023Updated 3 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…☆29Aug 8, 2020Updated 6 years ago
- SageMaker specific extensions to TensorFlow.☆54Jul 23, 2024Updated 2 years ago
- These are the Jupyter notebooks for the Big Data specialization in the Data Science Program.☆15Apr 3, 2020Updated 6 years ago
- Code from https://pythonwise.blogspot.com☆21Nov 23, 2023Updated 2 years ago
- Project files for the post: Running PySpark Applications on Amazon EMR using Apache Airflow: Using the new Amazon Managed Workflows for A…☆41Jul 6, 2022Updated 4 years ago
- Educational project on how to build an ETL (Extract, Transform, Load) data pipeline, orchestrated with Airflow.☆354Jan 12, 2022Updated 4 years ago
- My Git Repo for Csv Data☆21Oct 5, 2025Updated last year