jaumpedro214 / traffic-flow-spark-kafka
Testing Spark Structured Streaming anf Kafka with real data from traffic sensors
☆16Updated 2 years ago
Alternatives and similar repositories for traffic-flow-spark-kafka:
Users that are interested in traffic-flow-spark-kafka are comparing it to the libraries listed below
- Delta-Lake, ETL, Spark, Airflow☆47Updated 2 years ago
- ☆34Updated last year
- ☆40Updated 9 months ago
- Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO☆60Updated last year
- ☆38Updated 2 years ago
- A list of all my posts and personal projects☆70Updated 10 months ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆41Updated last year
- Essential PySpark for Scalable Data Analytics, published by Packt☆44Updated 2 years ago
- This is the repo of the Weather app from my YouTube video☆18Updated last year
- ☆28Updated last year
- Example repo to create end to end tests for data pipeline.☆23Updated 10 months ago
- ☆36Updated 2 years ago
- Processing TfL data for bike usage with Google Cloud Platform.☆45Updated 2 years ago
- A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…☆30Updated 4 years ago
- Template for data pipelines, ML workflows, API dev and monitoring☆45Updated last year
- MLOps for deploying a Credit Risk model☆30Updated last year
- Course Material Data Engineering on AWS Course☆28Updated 7 months ago
- A Postgres data warehouse for processing synthetic data using IAC principles☆17Updated 2 years ago
- code snippet for analytics sessions☆34Updated 2 years ago
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆67Updated last year
- Spark, Airflow, Kafka☆26Updated last year
- Data pipeline that scrapes Rust cheater Steam profiles☆52Updated 3 years ago
- Project for real-time anomaly detection using Kafka and python☆58Updated 2 years ago
- Simple ETL pipeline using Python☆26Updated last year
- End to end data engineering project☆54Updated 2 years ago
- Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks☆23Updated 3 years ago
- Data Engineering Capstone Project: ETL Pipelines and Data Warehouse Development☆21Updated 5 years ago
- reating a modern data pipeline using a combination of Terraform, AWS Lambda and S3, Snowflake, DBT, Mage AI, and Dash.☆14Updated last year
- Design/Implement stream/batch architecture on NYC taxi data | #DE☆25Updated 3 years ago
- ☆14Updated 2 years ago