pyjaime / docker-airflow-spark
Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks
☆23Updated 2 years ago
Alternatives and similar repositories for docker-airflow-spark:
Users that are interested in docker-airflow-spark are comparing it to the libraries listed below
- ☆36Updated 2 years ago
- Delta-Lake, ETL, Spark, Airflow☆46Updated 2 years ago
- ☆40Updated 8 months ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆40Updated last year
- End to end data engineering project☆53Updated 2 years ago
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆35Updated last year
- Building a Data Pipeline with an Open Source Stack☆49Updated 8 months ago
- A course by DataTalks Club that covers Spark, Kafka, Docker, Airflow, Terraform, DBT, Big Query etc☆14Updated 3 years ago
- Simple stream processing pipeline☆99Updated 9 months ago
- A workspace to experiment with Apache Spark, Livy, and Airflow in a Docker environment.☆38Updated 3 years ago
- Data pipeline that scrapes Rust cheater Steam profiles☆52Updated 3 years ago
- Docker with Airflow and Spark standalone cluster☆252Updated last year
- velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docke…☆18Updated 6 months ago
- ☆87Updated 2 years ago
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆65Updated last year
- Code snippets for Data Engineering Design Patterns book☆74Updated this week
- Near real time ETL to populate a dashboard.☆73Updated 9 months ago
- ☆27Updated last year
- Simple ETL pipeline using Python☆25Updated last year
- Code for my "Efficient Data Processing in SQL" book.☆56Updated 7 months ago
- In this article, you will learn how to set up a real-time data processing and analytics environment using Docker, MySQL, Redpanda, MinIO,…☆10Updated last year
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆136Updated last year
- Local Environment to Practice Data Engineering☆143Updated 2 months ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆86Updated 7 months ago
- ☆43Updated 2 years ago
- Project for "Data pipeline design patterns" blog.☆44Updated 7 months ago
- End-to-end Kafka Streaming Examples on Databricks with Evolving Avro Schemas.☆9Updated last year
- ☆10Updated 2 years ago
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆35Updated last year
- This project shows how to capture changes from postgres database and stream them into kafka☆36Updated 10 months ago