mtpatter / time-series-kafka-demoLinks
Fully reproducible, Dockerized, step-by-step, tutorial on how to mock a "real-time" Kafka data stream from a timestamped csv file. Detailed blog post published on Towards Data Science.
☆40Updated 3 years ago
Alternatives and similar repositories for time-series-kafka-demo
Users that are interested in time-series-kafka-demo are comparing it to the libraries listed below
Sorting:
- build dw with dbt☆47Updated last year
- ☆44Updated last year
- This project shows how to capture changes from postgres database and stream them into kafka☆38Updated last year
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆42Updated 2 years ago
- Some recipes for data engineering with Python☆23Updated 4 years ago
- A Series of Notebooks on how to start with Kafka and Python☆152Updated 8 months ago
- A Postgres data warehouse for processing synthetic data using IAC principles☆19Updated 2 years ago
- Code snippets for Data Engineering Design Patterns book☆262Updated 7 months ago
- Simple stream processing pipeline☆110Updated last year
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆45Updated last year
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆143Updated 2 years ago
- Apache Airflow advanced functionalities examples☆21Updated last year
- Full stack data engineering tools and infrastructure set-up☆57Updated 4 years ago
- Delta-Lake, ETL, Spark, Airflow☆48Updated 3 years ago
- used Airflow, Postgres, Kafka, Spark, and Cassandra, and GitHub Actions to establish an end-to-end data pipeline☆29Updated 2 years ago
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆42Updated last year
- Resources for video demonstrations and blog posts related to DataOps on AWS☆182Updated 3 years ago
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆37Updated 2 years ago
- Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks☆24Updated 3 years ago
- Series follows learning from Apache Spark (PySpark) with quick tips and workaround for daily problems in hand☆56Updated 2 years ago
- 📡 Real-time data pipeline with Kafka, Flink, Iceberg, Trino, MinIO, and Superset. Ideal for learning data systems.☆52Updated 9 months ago
- ☆88Updated 3 years ago
- Serverless ETL and Analytics with AWS Glue, published by Packt☆52Updated 2 years ago
- Apache Airflow Best Practices, published by Packt☆50Updated last year
- This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and …☆34Updated last year
- Docker Airflow - Contains a docker compose file for Airflow 2.0☆69Updated 3 years ago
- End to end data engineering project☆57Updated 3 years ago
- Data Engineering examples for Airflow, Prefect; dbt for BigQuery, Redshift, ClickHouse, Postgres, DuckDB; PySpark for Batch processing; K…☆68Updated 4 months ago
- This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging…☆94Updated 6 years ago
- A course by DataTalks Club that covers Spark, Kafka, Docker, Airflow, Terraform, DBT, Big Query etc☆14Updated 3 years ago