Automates the ETL process for recipe data stored in CSV files on GCP , transforming and loading the data into BigQuery tables. The pipeline includes data validation and transformation steps, and creates new tables in the "recipe_dataset" dataset for analysis and reporting
☆19May 20, 2023Updated 3 years ago
Alternatives and similar repositories for ETL-Pipeline-with-Airflow-and-BigQuery
Users that are interested in ETL-Pipeline-with-Airflow-and-BigQuery are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Fully automated csv to dashboard pipeline using Terraform, Google Cloud Storage, BigQuery, dbt, Prefect and Looker Studio. Peer ranked …☆52Nov 15, 2024Updated last year
- An end-to-end ETL pipeline that extracts weather data, transforms it, and loads it into a PostgreSQL database.☆15Sep 6, 2024Updated last year
- An ETL pipeline that extracts weather and air quality data from public APIs, transforms the data into a clean, analyzable format, and loa…☆49Sep 21, 2024Updated last year
- Features Databricks Apps examples that are built by Databricks field personnel. Meant to act as points-of-inspiration and points-of-imple…☆25Jan 19, 2026Updated 6 months ago
- ☆14Jun 3, 2022Updated 4 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Config files for my GitHub profile.☆22Jan 16, 2024Updated 2 years ago
- A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.☆18Jan 12, 2022Updated 4 years ago
- DevOps Kitchen workshop initiation offers practical Terraform tasks and a guide for beginners.☆23Jun 7, 2026Updated last month
- Transform data from on-premises SQL Server to Azure Delta Lake Storage for Analytics and Visualization☆27Jul 16, 2023Updated 3 years ago
- ☆47Feb 1, 2026Updated 5 months ago
- ☆35May 4, 2026Updated 2 months ago
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆82Aug 21, 2023Updated 2 years ago
- YouTube tutorial project☆109Oct 17, 2023Updated 2 years ago
- Distributed Matrix Library☆73Jan 28, 2017Updated 9 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Projects done in the Data Engineer Nanodegree Program by Udacity.com☆169Dec 8, 2022Updated 3 years ago
- Data Engineer Roadmap for 2024☆164Dec 10, 2025Updated 7 months ago
- Real-world AWS projects for aspiring cloud engineers — Beginner to Advanced. Covers AWS services, Infrastructure as Code, CI/CD, containe…☆109May 11, 2026Updated 2 months ago
- Practical Data Engineering: A Hands-On Real-Estate Project Guide☆815Jun 25, 2026Updated 3 weeks ago
- In this repository we store all materials for dlt workshops, courses, etc.☆260May 27, 2026Updated last month
- My 'Excel for Data Analytics' course on YouTube☆1,026Jul 9, 2026Updated last week
- An Awesome List of Open-Source Data Engineering Projects☆3,249Oct 4, 2024Updated last year
- This is a template you can use for your next data engineering portfolio project.☆191Sep 10, 2021Updated 4 years ago
- ☆193Nov 10, 2023Updated 2 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Public repo for 365 Data Science ML Algorithms Course☆285Jun 21, 2023Updated 3 years ago
- Topic Modeling on Apache Spark☆94Mar 1, 2019Updated 7 years ago
- ☆110Apr 17, 2017Updated 9 years ago
- A collection of AWS (Amazon Web Services) projects☆209Nov 16, 2024Updated last year
- All things awesome related to Dagster!☆147Jun 1, 2026Updated last month
- Personal Data Engineering Projects☆1,024Feb 8, 2023Updated 3 years ago
- A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for …☆138Apr 18, 2020Updated 6 years ago
- shareAI Lab's skills for agent to build agent & other custom software system☆303Updated this week
- Open-Source Kubernetes YAML Builder with Intuitive Web Interface and Dynamic Visualization for Developers and DevOps Engineers☆480Aug 16, 2025Updated 11 months ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- Beginner data engineering project - batch edition☆584Apr 13, 2026Updated 3 months ago
- ☆1,151Jan 31, 2026Updated 5 months ago
- End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world dat…☆366Jan 19, 2026Updated 6 months ago
- A framework for the evaluation of autoregressive code generation language models.☆1,052Jul 22, 2025Updated 11 months ago
- MLOps for DevOps Engineers - A hands-on, project-based guide to Machine Learning Operations☆464Updated this week
- ☆468Sep 9, 2025Updated 10 months ago
- Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake developme…☆1,953Aug 26, 2022Updated 3 years ago