skhatri / airflow-by-example
Bunch of Airflow Configurations and DAGs for Kubernetes, Spark based data-pipelines. Scale inside Kubernetes using spark kubernetes master. Secure it with keycloak
☆22Updated 2 years ago
Related projects ⓘ
Alternatives and complementary repositories for airflow-by-example
- A repository of sample code to show data quality checking best practices using Airflow.☆72Updated last year
- Learn how to add data validation and documentation to a data pipeline built with dbt and Airflow.☆166Updated last year
- (project & tutorial) dag pipeline tests + ci/cd setup☆85Updated 3 years ago
- Great Expectations Airflow operator☆159Updated 3 weeks ago
- Pylint plugin for static code analysis on Airflow code☆90Updated 4 years ago
- Astronomer Core Docker Images☆106Updated 6 months ago
- ☆20Updated 3 years ago
- A GitHub Action that makes it easy to use Great Expectations to validate your data pipelines in your CI workflows.☆80Updated 6 months ago
- Pipeline definitions for managing data flows to power analytics at MIT Open Learning☆38Updated this week
- Creates simple data models on Snowflake to report dbt source freshness and tests☆22Updated last year
- Snowflake Guide: Building a Recommendation Engine Using Snowflake & Amazon SageMaker☆31Updated 3 years ago
- Airflow Providers containing Deferrable Operators & Sensors from Astronomer☆139Updated last week
- locopy: Loading/Unloading to Redshift and Snowflake using Python.☆104Updated this week
- Rules based grant management for Snowflake☆40Updated 5 years ago
- triggering a DAG run multiple times☆85Updated 8 months ago
- Enforce Best Practices for all your Airflow DAGs. ⭐☆92Updated this week
- Make simple storing test results and visualisation of these in a BI dashboard☆40Updated last week
- The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and …☆83Updated 6 months ago
- Delta Lake Documentation☆47Updated 5 months ago
- Public source code for the Batch Processing with Apache Beam (Python) online course☆19Updated 4 years ago
- Delta Lake helper methods. No Spark dependency.☆22Updated 2 months ago
- A complete development environment setup for working with Airflow☆126Updated last year
- Airflow Backfill UI based plugin for existing / new Airflow environment☆66Updated 3 years ago
- ☆27Updated last year
- Airflow training for the crunch conf☆105Updated 6 years ago
- Fast iterative local development and testing of Apache Airflow workflows☆193Updated this week
- An example dbt project using AutomateDV to create a Data Vault 2.0 Data Warehouse based on the Snowflake TPC-H dataset.☆42Updated 8 months ago
- Dry run capability for dbt projects using BigQuery☆88Updated 4 months ago
- Pytest plugin for dbt core☆58Updated 5 months ago