anilkulkarni87 / airflow-dockerLinks
This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.
☆31Updated last year
Alternatives and similar repositories for airflow-docker
Users that are interested in airflow-docker are comparing it to the libraries listed below
Sorting:
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Updated last year
- ☆12Updated 3 years ago
- ☆12Updated 3 years ago
- Sample Airflow DAGs to load data from the CovidTracking API to Snowflake via an AWS S3 intermediary.☆16Updated 4 years ago
- reating a modern data pipeline using a combination of Terraform, AWS Lambda and S3, Snowflake, DBT, Mage AI, and Dash.☆14Updated last year
- Challenge Data Engineer☆25Updated 2 years ago
- Example repo to create end to end tests for data pipeline.☆24Updated 11 months ago
- PySpark Cheatsheet☆36Updated 2 years ago
- Full stack data engineering tools and infrastructure set-up☆53Updated 4 years ago
- To provide a deeper understanding of how the modern, open-source data stack consisting of Iceberg, dbt, Trino, and Hive operates within a…☆35Updated last year
- ☆40Updated 11 months ago
- The Data Pipeline and Analytics Stack is a comprehensive solution designed for processing, storing, and visualizing data. Explore a compl…☆14Updated last year
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆16Updated 5 years ago
- build dw with dbt☆45Updated 7 months ago
- Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validatio…☆55Updated 2 years ago
- ☆38Updated 2 years ago
- Simple ETL pipeline using Python☆26Updated 2 years ago
- Data pipeline that scrapes Rust cheater Steam profiles☆51Updated 3 years ago
- End-to-end ELT data engineering project☆21Updated 2 years ago
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆69Updated last year
- Deploying a Machine Learning model streaming application with Apache Kafka☆10Updated 2 years ago
- A Postgres data warehouse for processing synthetic data using IAC principles☆17Updated 2 years ago
- Demo on how to use Prefect 2 in an ML project☆41Updated 2 years ago
- Design/Implement stream/batch architecture on NYC taxi data | #DE☆25Updated 4 years ago
- End to end data engineering project☆56Updated 2 years ago
- A data pipeline moving data from a Relational database system (RDBMS) to a Hadoop file system (HDFS).☆15Updated 4 years ago
- Public source code for the Batch Processing with Apache Beam (Python) online course☆18Updated 4 years ago
- pyspark dataframe made easy☆16Updated 3 years ago
- This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which…☆98Updated 10 months ago
- Project for real-time anomaly detection using Kafka and python☆58Updated 2 years ago