anilkulkarni87 / airflow-docker
This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.
☆29Updated last year
Alternatives and similar repositories for airflow-docker
Users that are interested in airflow-docker are comparing it to the libraries listed below
Sorting:
- Delta-Lake, ETL, Spark, Airflow☆47Updated 2 years ago
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Updated last year
- Finance 🏦 Data Builder 🛠️ @ postgres 🐘☆21Updated 4 years ago
- To provide a deeper understanding of how the modern, open-source data stack consisting of Iceberg, dbt, Trino, and Hive operates within a…☆35Updated last year
- Airflow Tutorials☆25Updated 4 years ago
- Big Data Demystified meetup and blog examples☆31Updated 9 months ago
- Project for real-time anomaly detection using Kafka and python☆59Updated 2 years ago
- Public source code for the Batch Processing with Apache Beam (Python) online course☆18Updated 4 years ago
- Building a Data Pipeline with an Open Source Stack☆54Updated 10 months ago
- A portable Datamart and Business Intelligence suite built with Docker, Airflow, dbt, PostgreSQL and Superset☆41Updated 6 months ago
- Full stack data engineering tools and infrastructure set-up☆52Updated 4 years ago
- The Data Pipeline and Analytics Stack is a comprehensive solution designed for processing, storing, and visualizing data. Explore a compl…☆13Updated last year
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆68Updated last year
- Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks☆23Updated 3 years ago
- dlt-dagster-demo☆11Updated last year
- reating a modern data pipeline using a combination of Terraform, AWS Lambda and S3, Snowflake, DBT, Mage AI, and Dash.☆14Updated last year
- Docker compose and Google Colab demo to build a CDC with Delta Lake☆15Updated 2 years ago
- ☆12Updated 3 years ago
- Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validatio…☆55Updated 2 years ago
- Code and notebooks containing my experiments in data science, EDA, visualization, and machine learning☆27Updated last year
- Creation of a data lakehouse and an ELT pipeline to enable the efficient analysis and use of data☆46Updated last year
- PySpark Cheatsheet☆36Updated 2 years ago
- Data pipeline that scrapes Rust cheater Steam profiles☆52Updated 3 years ago
- Series follows learning from Apache Spark (PySpark) with quick tips and workaround for daily problems in hand☆51Updated last year
- Demo on how to use Prefect 2 in an ML project☆41Updated 2 years ago
- A Series of Notebooks on how to start with Kafka and Python☆154Updated 2 months ago
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆29Updated last year
- build dw with dbt☆44Updated 6 months ago
- Streamlit application to explore Snowflake Tables☆40Updated last year
- Sample Airflow DAGs to load data from the CovidTracking API to Snowflake via an AWS S3 intermediary.☆16Updated 4 years ago