nishakanthiA / Airflow-Docker
Orchestration of data science models in Apache Airflow, scale-up with Celery Executor and deploying in multiple Docker containers
☆31Updated 4 years ago
Alternatives and similar repositories for Airflow-Docker:
Users that are interested in Airflow-Docker are comparing it to the libraries listed below
- Delta-Lake, ETL, Spark, Airflow☆46Updated 2 years ago
- ☆36Updated 2 years ago
- Local development environment for python data projects, with Docker☆21Updated 2 years ago
- Resources for video demonstrations and blog posts related to DataOps on AWS☆172Updated 3 years ago
- Docker with Airflow and Spark standalone cluster☆251Updated last year
- (project & tutorial) dag pipeline tests + ci/cd setup☆86Updated 4 years ago
- Project for "Data pipeline design patterns" blog.☆44Updated 7 months ago
- Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks☆23Updated 2 years ago
- ☆40Updated 8 months ago
- End to end data engineering project☆53Updated 2 years ago
- Processing TfL data for bike usage with Google Cloud Platform.☆45Updated 2 years ago
- ☆181Updated 2 years ago
- A Series of Notebooks on how to start with Kafka and Python☆154Updated 3 weeks ago
- Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)☆183Updated last year
- This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging…☆78Updated 5 years ago
- Code snippets for Data Engineering Design Patterns book☆74Updated last month
- An example of a Dagster project with a possible folder structure to organize the assets, jobs, repositories, schedules, and ops. Also has…☆92Updated 4 months ago
- Execution of DBT models using Apache Airflow through Docker Compose☆115Updated 2 years ago
- Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)☆289Updated last year
- A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…☆30Updated 4 years ago
- Example repo to create end to end tests for data pipeline.☆22Updated 9 months ago
- Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'☆119Updated last year
- Near real time ETL to populate a dashboard.☆73Updated 9 months ago
- ☆37Updated last year
- A workspace to experiment with Apache Spark, Livy, and Airflow in a Docker environment.☆39Updated 3 years ago
- Learn how to add data validation and documentation to a data pipeline built with dbt and Airflow.☆167Updated last year
- Code for my "Efficient Data Processing in SQL" book.☆56Updated 7 months ago
- The resources of the preparation course for Databricks Data Engineer Professional certification exam☆108Updated last month
- Docker Airflow - Contains a docker compose file for Airflow 2.0☆65Updated 2 years ago