pyjaime / docker-airflow-spark
Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks
☆23Updated 2 years ago
Alternatives and similar repositories for docker-airflow-spark:
Users that are interested in docker-airflow-spark are comparing it to the libraries listed below
- ☆35Updated 2 years ago
- Delta-Lake, ETL, Spark, Airflow☆46Updated 2 years ago
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆65Updated last year
- ☆41Updated 8 months ago
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆35Updated last year
- Code snippets for Data Engineering Design Patterns book☆74Updated last month
- A course by DataTalks Club that covers Spark, Kafka, Docker, Airflow, Terraform, DBT, Big Query etc☆14Updated 2 years ago
- Simple stream processing pipeline☆99Updated 8 months ago
- Docker with Airflow and Spark standalone cluster☆251Updated last year
- Building a Modern Data Lake with Minio, Spark, Airflow via Docker.☆16Updated 10 months ago
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆136Updated last year
- A workspace to experiment with Apache Spark, Livy, and Airflow in a Docker environment.☆39Updated 3 years ago
- End to end data engineering project☆53Updated 2 years ago
- ☆15Updated last year
- ☆87Updated 2 years ago
- velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docke…☆18Updated 6 months ago
- Essential PySpark for Scalable Data Analytics, published by Packt☆43Updated 2 years ago
- Project for "Data pipeline design patterns" blog.☆44Updated 7 months ago
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆34Updated last year
- ☆21Updated this week
- Code for dbt tutorial☆153Updated 9 months ago
- Generate synthetic Spotify music stream dataset to create dashboards. Spotify API generates fake event data emitted to Kafka. Spark consu…☆67Updated last year
- Code for blog at: https://www.startdataengineering.com/post/docker-for-de/☆34Updated 10 months ago
- Code for my "Efficient Data Processing in SQL" book.☆56Updated 7 months ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆38Updated last year
- Delta Lake examples☆218Updated 5 months ago
- Apache Airflow Best Practices, published by Packt☆37Updated 4 months ago
- Local Environment to Practice Data Engineering☆142Updated 2 months ago
- Example repo to create end to end tests for data pipeline.☆22Updated 9 months ago
- A list of all my posts and personal projects☆70Updated 9 months ago