jaumpedro214 / traffic-flow-spark-kafkaLinks
Testing Spark Structured Streaming anf Kafka with real data from traffic sensors
☆17Updated 3 years ago
Alternatives and similar repositories for traffic-flow-spark-kafka
Users that are interested in traffic-flow-spark-kafka are comparing it to the libraries listed below
Sorting:
- ☆37Updated 6 years ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆43Updated 2 years ago
- Mastering Big Data Analytics with PySpark, Published by Packt☆165Updated last year
- Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO☆65Updated 2 years ago
- Apache Spark using SQL☆14Updated 4 years ago
- ☆12Updated 4 years ago
- This project aims to build a streaming application to perform real-time analytics of Covid-19 related tweets and deploy an ML model for r…☆15Updated 4 years ago
- Stream/batch system with Hadoop, Spark on NYC taxi data | #DE☆26Updated 4 months ago
- Deploying Models to Production with Mlflow and AWS Sagemaker☆24Updated 4 years ago
- A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract d…☆24Updated 4 years ago
- A list of all my posts and personal projects☆73Updated 3 months ago
- Project for "Data pipeline design patterns" blog.☆50Updated last year
- ☆88Updated 3 years ago
- Data Engineering Capstone Project: ETL Pipelines and Data Warehouse Development☆21Updated 6 years ago
- Simple ETL pipeline using Python☆29Updated 2 years ago
- ☆45Updated last year
- RedditR for Content Engagement and Recommendation☆18Updated 8 years ago
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆144Updated 2 years ago
- PySpark functions and utilities with examples. Assists ETL process of data modeling☆104Updated 5 years ago
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆16Updated 6 years ago
- The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such …☆123Updated 3 years ago
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆38Updated 2 years ago
- This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and …☆34Updated last year
- Spark, Airflow, Kafka☆24Updated 2 years ago
- Duke MIDS: Data Engineering and DataOps Course☆69Updated last year
- Project for real-time anomaly detection using Kafka and python☆59Updated 3 years ago
- ☆21Updated 2 years ago
- Processing TfL data for bike usage with Google Cloud Platform.☆46Updated 3 years ago
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆45Updated 2 years ago
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆47Updated 2 years ago