Automates the ETL process for recipe data stored in CSV files on GCP , transforming and loading the data into BigQuery tables. The pipeline includes data validation and transformation steps, and creates new tables in the "recipe_dataset" dataset for analysis and reporting
☆21May 20, 2023Updated 3 years ago
Alternatives and similar repositories for ETL-Pipeline-with-Airflow-and-BigQuery
Users that are interested in ETL-Pipeline-with-Airflow-and-BigQuery are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Fully automated csv to dashboard pipeline using Terraform, Google Cloud Storage, BigQuery, dbt, Prefect and Looker Studio. Peer ranked …☆54Nov 15, 2024Updated last year
- An end-to-end ETL pipeline that extracts weather data, transforms it, and loads it into a PostgreSQL database.☆16Sep 6, 2024Updated last year
- Features Databricks Apps examples that are built by Databricks field personnel. Meant to act as points-of-inspiration and points-of-imple…☆26Jan 19, 2026Updated 7 months ago
- ☆15Jun 26, 2025Updated last year
- ☆14Jun 3, 2022Updated 4 years ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- Config files for my GitHub profile.☆22Jan 16, 2024Updated 2 years ago
- A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.☆18Jan 12, 2022Updated 4 years ago
- ☆19Jan 28, 2026Updated 7 months ago
- ☆18Feb 16, 2022Updated 4 years ago
- DevOps Kitchen workshop initiation offers practical Terraform tasks and a guide for beginners.☆24Jul 27, 2026Updated last month
- Transform data from on-premises SQL Server to Azure Delta Lake Storage for Analytics and Visualization☆29Jul 16, 2023Updated 3 years ago
- ☆56Feb 1, 2026Updated 6 months ago
- Code for the second edition of Data Pipelines with Apache Airflow Book☆56Feb 11, 2026Updated 6 months ago
- End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)☆40Aug 9, 2024Updated 2 years ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆82Aug 21, 2023Updated 3 years ago
- Site Reliability Engineering Cheat Sheet☆18May 19, 2020Updated 6 years ago
- This repo is for Linkedin Learning course: Prometheus Essential Training☆30Apr 12, 2024Updated 2 years ago
- YouTube tutorial project☆110Oct 17, 2023Updated 2 years ago
- This article compares open-source Python packages for pipeline/workflow development: Airflow, Luigi, Gokart, Metaflow, Kedro, PipelineX.☆56Jul 10, 2020Updated 6 years ago
- Projects done in the Data Engineer Nanodegree Program by Udacity.com☆171Dec 8, 2022Updated 3 years ago
- Real-world AWS projects for aspiring cloud engineers — Beginner to Advanced. Covers AWS services, Infrastructure as Code, CI/CD, containe…☆130May 11, 2026Updated 3 months ago
- Practical Data Engineering: A Hands-On Real-Estate Project Guide☆821Jun 25, 2026Updated 2 months ago
- In this repository we store all materials for dlt workshops, courses, etc.☆260May 27, 2026Updated 3 months ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Repo for Relational Databases video☆110Jan 1, 2025Updated last year
- Simple downloaded for https://kodekloud.com/☆244Jun 9, 2026Updated 2 months ago
- Public repo for 365 Data Science ML Algorithms Course☆289Jun 21, 2023Updated 3 years ago
- A collection of AWS (Amazon Web Services) projects☆220Nov 16, 2024Updated last year
- All things awesome related to Dagster!☆147Jun 1, 2026Updated 2 months ago
- Personal Data Engineering Projects☆1,031Feb 8, 2023Updated 3 years ago
- A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for …☆138Apr 18, 2020Updated 6 years ago
- A MCP (Model Context Protocol) server for interacting with dbt.☆601Updated this week
- DevOps Interview Questions and answers for juniors☆398May 6, 2024Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Skills distilled from the Lab's real work and collaboration practices☆311Updated this week
- Databricks Toolkit for Coding Agents provided by Field Engineering☆1,880Aug 13, 2026Updated 2 weeks ago
- Open-Source Kubernetes YAML Builder with Intuitive Web Interface and Dynamic Visualization for Developers and DevOps Engineers☆480Aug 16, 2025Updated last year
- Beginner data engineering project - batch edition☆591Apr 13, 2026Updated 4 months ago
- ☆1,247Jan 31, 2026Updated 7 months ago
- End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world dat…☆399Jan 19, 2026Updated 7 months ago
- A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.☆921Apr 23, 2025Updated last year