jaumpedro214 / traffic-flow-spark-kafkaLinks
Testing Spark Structured Streaming anf Kafka with real data from traffic sensors
☆16Updated 2 years ago
Alternatives and similar repositories for traffic-flow-spark-kafka
Users that are interested in traffic-flow-spark-kafka are comparing it to the libraries listed below
Sorting:
- Base Kafka Producer, consumer, flask api and PySpark Structured streaming Job☆11Updated 3 years ago
- ☆37Updated 5 years ago
- ☆44Updated last year
- Project for real-time anomaly detection using Kafka and python☆58Updated 2 years ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆42Updated last year
- This project aims to build a streaming application to perform real-time analytics of Covid-19 related tweets and deploy an ML model for r…☆14Updated 4 years ago
- Design/Implement stream/batch architecture on NYC taxi data | #DE☆25Updated 4 years ago
- Processing TfL data for bike usage with Google Cloud Platform.☆45Updated 3 years ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆24Updated 3 years ago
- A list of all my posts and personal projects☆74Updated last year
- Course Material Data Engineering on AWS Course☆29Updated 11 months ago
- Mastering Big Data Analytics with PySpark, Published by Packt☆161Updated last year
- Spark, Airflow, Kafka☆26Updated 2 years ago
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆37Updated 2 years ago
- Apache Spark using SQL☆14Updated 4 years ago
- Deploying Models to Production with Mlflow and AWS Sagemaker☆23Updated 3 years ago
- Series follows learning from Apache Spark (PySpark) with quick tips and workaround for daily problems in hand☆55Updated last year
- used Airflow, Postgres, Kafka, Spark, and Cassandra, and GitHub Actions to establish an end-to-end data pipeline☆29Updated last year
- Delta-Lake, ETL, Spark, Airflow☆48Updated 2 years ago
- A Series of Notebooks on how to start with Kafka and Python☆152Updated 6 months ago
- MLOps for deploying a Credit Risk model☆32Updated 2 years ago
- A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract d…☆24Updated 3 years ago
- Repository for Data Engineering Interview Series☆31Updated 10 months ago
- Classwork projects and home works done through Udacity data engineering nano degree☆74Updated last year
- reating a modern data pipeline using a combination of Terraform, AWS Lambda and S3, Snowflake, DBT, Mage AI, and Dash.☆14Updated 2 years ago
- This is an overview of a MLOps architecture that includes both Airflow and MLflow running on separate Docker containers.☆21Updated 2 years ago
- Simple demo for Databricks!☆14Updated last year
- Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO☆63Updated 2 years ago
- PySpark functions and utilities with examples. Assists ETL process of data modeling☆104Updated 4 years ago
- ☆40Updated 2 years ago