mtpatter / time-series-kafka-demoLinks
Fully reproducible, Dockerized, step-by-step, tutorial on how to mock a "real-time" Kafka data stream from a timestamped csv file. Detailed blog post published on Towards Data Science.
☆40Updated 4 years ago
Alternatives and similar repositories for time-series-kafka-demo
Users that are interested in time-series-kafka-demo are comparing it to the libraries listed below
Sorting:
- build dw with dbt☆50Updated last year
- Resources for video demonstrations and blog posts related to DataOps on AWS☆182Updated 4 years ago
- Code snippets for Data Engineering Design Patterns book☆317Updated 3 weeks ago
- Some recipes for data engineering with Python☆25Updated 4 years ago
- Apache Airflow advanced functionalities examples☆21Updated last year
- A Series of Notebooks on how to start with Kafka and Python☆151Updated 11 months ago
- Simple stream processing pipeline☆110Updated last year
- Delta Lake Documentation☆53Updated last year
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆144Updated 2 years ago
- Full stack data engineering tools and infrastructure set-up☆57Updated 4 years ago
- Docker Airflow - Contains a docker compose file for Airflow 2.0☆70Updated 3 years ago
- Code for dbt tutorial☆166Updated 4 months ago
- Apache Airflow Best Practices, published by Packt☆50Updated last year
- A Postgres data warehouse for processing synthetic data using IAC principles☆19Updated 2 years ago
- Delta-Lake, ETL, Spark, Airflow☆48Updated 3 years ago
- ☆45Updated last year
- A course by DataTalks Club that covers Spark, Kafka, Docker, Airflow, Terraform, DBT, Big Query etc☆15Updated 3 years ago
- This project focuses on building a robust data pipeline using Apache Airflow to automate the ingestion of weather data from the OpenWeath…☆21Updated 2 years ago
- This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging…☆96Updated 6 years ago
- Data Engineering examples for Airflow, Prefect; dbt for BigQuery, Redshift, ClickHouse, Postgres, DuckDB; PySpark for Batch processing; K…☆69Updated this week
- Productionalizing Data Pipelines with Apache Airflow☆116Updated 3 years ago
- Building a Data Pipeline with an Open Source Stack☆55Updated 7 months ago
- Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks☆24Updated 3 years ago
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Updated 4 years ago
- Code for blog at: https://www.startdataengineering.com/post/docker-for-de/☆40Updated last year
- A handpicked collection of resources for Python developers in data engineering, machine learning, and AI. Inside, you'll discover a neatl…☆122Updated last year
- This is a demo streaming project simulating a music streaming service.☆34Updated last year
- Streaming Synthetic Sales Data Generator: Streaming sales data generator for Apache Kafka, written in Python☆44Updated 3 years ago
- A workspace to experiment with Apache Spark, Livy, and Airflow in a Docker environment.☆38Updated 4 years ago
- End-to-end data platform leveraging the Modern data stack☆52Updated last year