jaumpedro214 / traffic-flow-spark-kafkaLinks
Testing Spark Structured Streaming anf Kafka with real data from traffic sensors
☆16Updated 2 years ago
Alternatives and similar repositories for traffic-flow-spark-kafka
Users that are interested in traffic-flow-spark-kafka are comparing it to the libraries listed below
Sorting:
- ☆37Updated 5 years ago
- Base Kafka Producer, consumer, flask api and PySpark Structured streaming Job☆11Updated 3 years ago
- Design/Implement stream/batch architecture on NYC taxi data | #DE☆25Updated 4 years ago
- A list of all my posts and personal projects☆73Updated last year
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆41Updated last year
- Kafka variant of the MLOps Level 1 stack☆25Updated 3 years ago
- A Series of Notebooks on how to start with Kafka and Python☆154Updated 4 months ago
- used Airflow, Postgres, Kafka, Spark, and Cassandra, and GitHub Actions to establish an end-to-end data pipeline☆28Updated last year
- ☆41Updated 11 months ago
- Project for real-time anomaly detection using Kafka and python☆57Updated 2 years ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆42Updated last year
- Course Material Data Engineering on AWS Course☆29Updated 9 months ago
- ☆39Updated 2 years ago
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆32Updated last year
- Simple stream processing pipeline☆102Updated last year
- ☆21Updated 2 years ago
- This project shows how to capture changes from postgres database and stream them into kafka☆36Updated last year
- Example repo to create end to end tests for data pipeline.☆25Updated last year
- Series follows learning from Apache Spark (PySpark) with quick tips and workaround for daily problems in hand☆55Updated last year
- End to end data engineering project☆56Updated 2 years ago
- Deploying Models to Production with Mlflow and AWS Sagemaker☆22Updated 3 years ago
- Repository for Data Engineering Interview Series☆32Updated 8 months ago
- Simple ETL pipeline using Python☆26Updated 2 years ago
- ☆16Updated last year
- PySpark functions and utilities with examples. Assists ETL process of data modeling☆103Updated 4 years ago
- A Postgres data warehouse for processing synthetic data using IAC principles☆18Updated 2 years ago
- Project for "Data pipeline design patterns" blog.☆45Updated 10 months ago
- An example of an ETL pipeline that lays out generic DE processes. This is now out of date but still provides useful information☆26Updated 3 years ago
- This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and …☆31Updated last year
- Spark application to consume kafka events generated by a python producer.☆12Updated 3 years ago