chayansraj / Data-Pipeline-with-dbt-using-Airflow-on-GCP
This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. There are different tools that have been used in this project such as Astro, DBT, GCP, Airflow, Metabase.
☆18Updated last week
Related projects: ⓘ
- Code for dbt tutorial☆138Updated 3 months ago
- Data lake, data warehouse on GCP☆54Updated 2 years ago
- Sample project to demonstrate data engineering best practices☆156Updated 6 months ago
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆127Updated 4 years ago
- End to end data engineering project☆49Updated last year
- Cloned by the `dbt init` task☆58Updated 4 months ago
- ☆14Updated last year
- A template repository to create a data project with IAC, CI/CD, Data migrations, & testing☆225Updated 2 months ago
- ☆102Updated last month
- ☆123Updated last year
- Simple stream processing pipeline☆89Updated 3 months ago
- Execution of DBT models using Apache Airflow through Docker Compose☆111Updated last year
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆54Updated last year
- This project is for demonstrating knowledge of Data Engineering tools and concepts and also learning in the process☆44Updated last year
- ☆35Updated 2 months ago
- Data engineering with dbt, published by Packt☆55Updated 6 months ago
- Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.☆191Updated last year
- Code for blog at https://www.startdataengineering.com/post/python-for-de/☆47Updated 3 months ago
- Resources for video demonstrations and blog posts related to DataOps on AWS☆166Updated 2 years ago
- build dw with dbt☆26Updated last month
- ☆124Updated 2 years ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆36Updated 11 months ago
- PySpark functions and utilities with examples. Assists ETL process of data modeling☆98Updated 3 years ago
- Code for my "Efficient Data Processing in SQL" book.☆47Updated last month
- Data Engineering examples covering Airflow and Mage for workflows; dbt for BigQuery, Redshift, ClickHouse; Spark and Kafka for Batch/Stre…☆50Updated 3 weeks ago
- A repository of sample code to show data quality checking best practices using Airflow.☆71Updated last year
- Project for "Data pipeline design patterns" blog.☆41Updated last month
- Delta-Lake, ETL, Spark, Airflow☆42Updated last year
- Near real time ETL to populate a dashboard.☆69Updated 3 months ago
- Course notes for the Astronomer Certification DAG Authoring for Apache Airflow☆44Updated 5 months ago