chayansraj / Data-Pipeline-with-dbt-using-Airflow-on-GCP
This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. There are different tools that have been used in this project such as Astro, DBT, GCP, Airflow, Metabase.
☆21Updated last week
Alternatives and similar repositories for Data-Pipeline-with-dbt-using-Airflow-on-GCP:
Users that are interested in Data-Pipeline-with-dbt-using-Airflow-on-GCP are comparing it to the libraries listed below
- Cloned by the `dbt init` task☆61Updated 11 months ago
- Code for dbt tutorial☆153Updated 9 months ago
- Execution of DBT models using Apache Airflow through Docker Compose☆116Updated 2 years ago
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆66Updated last year
- ⚙️ Airflow data pipeline with Terraform, GCP BigQuery, dbt, Soda and Looker Studio.☆21Updated last year
- Simple stream processing pipeline☆99Updated 9 months ago
- A modern ELT demo using airbyte, dbt, snowflake and dagster☆27Updated 2 years ago
- Delta-Lake, ETL, Spark, Airflow☆46Updated 2 years ago
- Resources for video demonstrations and blog posts related to DataOps on AWS☆172Updated 3 years ago
- End to end data engineering project☆53Updated 2 years ago
- Data lake, data warehouse on GCP☆56Updated 3 years ago
- Sample project to demonstrate data engineering best practices☆184Updated last year
- ☆126Updated last month
- Streaming Synthetic Sales Data Generator: Streaming sales data generator for Apache Kafka, written in Python☆43Updated 2 years ago
- An example of an ETL pipeline that lays out generic DE processes. This is now out of date but still provides useful information☆27Updated 2 years ago
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆141Updated 4 years ago
- PySpark functions and utilities with examples. Assists ETL process of data modeling☆100Updated 4 years ago
- Code snippets for Data Engineering Design Patterns book☆75Updated last week
- A template repository to create a data project with IAC, CI/CD, Data migrations, & testing☆260Updated 8 months ago
- Materials for the next course☆24Updated 2 years ago
- Project files for the post: Running PySpark Applications on Amazon EMR using Apache Airflow: Using the new Amazon Managed Workflows for A…☆40Updated 2 years ago
- build dw with dbt☆43Updated 5 months ago
- A portable Datamart and Business Intelligence suite built with Docker, Airflow, dbt, PostgreSQL and Superset☆39Updated 4 months ago
- Repo for CDC with debezium blog post☆28Updated 6 months ago
- Docker with Airflow and Spark standalone cluster☆253Updated last year
- Data engineering with dbt, published by Packt☆76Updated last year
- Data Engineering examples for Airflow, Prefect; dbt for BigQuery, Redshift, ClickHouse, Postgres, DuckDB; PySpark for Batch processing; K…☆64Updated last month
- A tutorial for the Great Expectations library.☆69Updated 4 years ago
- A Python package that creates fine-grained dbt tasks on Apache Airflow☆65Updated 6 months ago
- ☆20Updated 3 years ago