Automates the ETL process for recipe data stored in CSV files on GCP , transforming and loading the data into BigQuery tables. The pipeline includes data validation and transformation steps, and creates new tables in the "recipe_dataset" dataset for analysis and reporting
☆20May 20, 2023Updated 3 years ago
Alternatives and similar repositories for ETL-Pipeline-with-Airflow-and-BigQuery
Users that are interested in ETL-Pipeline-with-Airflow-and-BigQuery are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Fully automated csv to dashboard pipeline using Terraform, Google Cloud Storage, BigQuery, dbt, Prefect and Looker Studio. Peer ranked …☆53Nov 15, 2024Updated last year
- An end-to-end ETL pipeline that extracts weather data, transforms it, and loads it into a PostgreSQL database.☆15Sep 6, 2024Updated last year
- ☆15Jun 26, 2025Updated last year
- A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.☆18Jan 12, 2022Updated 4 years ago
- ☆19Jan 28, 2026Updated 6 months ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Best practices for data workflows, integrations with the Modern Data Stack (MDS), Infrastructure as Code (IaC), Cloud Provider Services☆38Jan 21, 2026Updated 6 months ago
- DevOps Kitchen workshop initiation offers practical Terraform tasks and a guide for beginners.☆24Jul 27, 2026Updated 2 weeks ago
- Transform data from on-premises SQL Server to Azure Delta Lake Storage for Analytics and Visualization☆28Jul 16, 2023Updated 3 years ago
- ☆49Feb 1, 2026Updated 6 months ago
- Code for the second edition of Data Pipelines with Apache Airflow Book☆55Feb 11, 2026Updated 5 months ago
- ☆35May 4, 2026Updated 3 months ago
- End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)☆40Aug 9, 2024Updated 2 years ago
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆82Aug 21, 2023Updated 2 years ago
- gokart file manager☆26Jul 28, 2026Updated 2 weeks ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- YouTube tutorial project☆109Oct 17, 2023Updated 2 years ago
- A curated list of awesome Harbor ecosystem projects☆52May 29, 2026Updated 2 months ago
- Machine learning tasks which are used with data pipeline library "luigi" and its wrapper "gokart".☆44Nov 25, 2023Updated 2 years ago
- This article compares open-source Python packages for pipeline/workflow development: Airflow, Luigi, Gokart, Metaflow, Kedro, PipelineX.☆56Jul 10, 2020Updated 6 years ago
- Projects done in the Data Engineer Nanodegree Program by Udacity.com☆170Dec 8, 2022Updated 3 years ago
- Practical Data Engineering: A Hands-On Real-Estate Project Guide☆816Jun 25, 2026Updated last month
- In this repository we store all materials for dlt workshops, courses, etc.☆259May 27, 2026Updated 2 months ago
- This repository contains a collection of SQL scripts demonstrating various analytical techniques, such as changes over time, cumulative, …☆251Mar 26, 2025Updated last year
- An Awesome List of Open-Source Data Engineering Projects☆3,265Oct 4, 2024Updated last year
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- This is a template you can use for your next data engineering portfolio project.☆192Sep 10, 2021Updated 4 years ago
- ☆196Nov 10, 2023Updated 2 years ago
- Topic Modeling on Apache Spark☆94Mar 1, 2019Updated 7 years ago
- ☆110Apr 17, 2017Updated 9 years ago
- Personal Data Engineering Projects☆1,024Feb 8, 2023Updated 3 years ago
- Claude Code Commands Manager☆313Mar 13, 2026Updated 4 months ago
- A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for …☆138Apr 18, 2020Updated 6 years ago
- A MCP (Model Context Protocol) server for interacting with dbt.☆595Updated this week
- DevOps Interview Questions and answers for juniors☆395May 6, 2024Updated 2 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- shareAI Lab's skills for agent to build agent & other custom software system☆307Jul 24, 2026Updated 2 weeks ago
- Open-Source Kubernetes YAML Builder with Intuitive Web Interface and Dynamic Visualization for Developers and DevOps Engineers☆479Aug 16, 2025Updated 11 months ago
- Beginner data engineering project - batch edition☆588Apr 13, 2026Updated 3 months ago
- ☆1,198Jan 31, 2026Updated 6 months ago
- End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world dat…☆381Jan 19, 2026Updated 6 months ago
- A framework for the evaluation of autoregressive code generation language models.☆1,055Jul 22, 2025Updated last year
- PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more☆228Apr 15, 2026Updated 3 months ago