Automates the ETL process for recipe data stored in CSV files on GCP , transforming and loading the data into BigQuery tables. The pipeline includes data validation and transformation steps, and creates new tables in the "recipe_dataset" dataset for analysis and reporting
☆21May 20, 2023Updated 3 years ago
Alternatives and similar repositories for ETL-Pipeline-with-Airflow-and-BigQuery
Users that are interested in ETL-Pipeline-with-Airflow-and-BigQuery are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Fully automated csv to dashboard pipeline using Terraform, Google Cloud Storage, BigQuery, dbt, Prefect and Looker Studio. Peer ranked …☆54Nov 15, 2024Updated last year
- An ETL pipeline that extracts weather and air quality data from public APIs, transforms the data into a clean, analyzable format, and loa…☆53Sep 21, 2024Updated last year
- ☆15Jun 26, 2025Updated last year
- ☆14Jun 3, 2022Updated 4 years ago
- Config files for my GitHub profile.☆22Jan 16, 2024Updated 2 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.☆18Jan 12, 2022Updated 4 years ago
- ☆19Jan 28, 2026Updated 7 months ago
- DevOps Kitchen workshop initiation offers practical Terraform tasks and a guide for beginners.☆24Sep 9, 2026Updated last week
- Transform data from on-premises SQL Server to Azure Delta Lake Storage for Analytics and Visualization☆29Jul 16, 2023Updated 3 years ago
- ☆60Feb 1, 2026Updated 7 months ago
- ☆37May 4, 2026Updated 4 months ago
- End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)☆41Aug 9, 2024Updated 2 years ago
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆84Aug 21, 2023Updated 3 years ago
- ☆67Sep 5, 2020Updated 6 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- YouTube tutorial project☆112Oct 17, 2023Updated 2 years ago
- Spark MLlib code optimized to efficiently support sparse data☆50Dec 22, 2016Updated 9 years ago
- Machine learning tasks which are used with data pipeline library "luigi" and its wrapper "gokart".☆44Nov 25, 2023Updated 2 years ago
- This article compares open-source Python packages for pipeline/workflow development: Airflow, Luigi, Gokart, Metaflow, Kedro, PipelineX.☆56Jul 10, 2020Updated 6 years ago
- Distributed Matrix Library☆73Jan 28, 2017Updated 9 years ago
- Projects done in the Data Engineer Nanodegree Program by Udacity.com☆171Dec 8, 2022Updated 3 years ago
- Documentation and Developer resources for Layer5 products☆99Updated this week
- Practical Data Engineering: A Hands-On Real-Estate Project Guide☆825Jun 25, 2026Updated 2 months ago
- My 'Excel for Data Analytics' course on YouTube☆1,137Sep 9, 2026Updated last week
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- An Awesome List of Open-Source Data Engineering Projects☆3,295Oct 4, 2024Updated last year
- Repo for Relational Databases video☆111Jan 1, 2025Updated last year
- This is a template you can use for your next data engineering portfolio project.☆194Sep 10, 2021Updated 5 years ago
- ☆202Nov 10, 2023Updated 2 years ago
- JimShapedCoding Python Crash Course 2021☆81Jun 7, 2022Updated 4 years ago
- Simple downloaded for https://kodekloud.com/☆244Jun 9, 2026Updated 3 months ago
- Topic Modeling on Apache Spark☆94Mar 1, 2019Updated 7 years ago
- ☆110Apr 17, 2017Updated 9 years ago
- Welcome to data analysis with pandas tutorial. In this tutorial i have covered all the topic of pandas and tried to explain with lesser n…☆82Oct 17, 2020Updated 5 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- A collection of AWS (Amazon Web Services) projects☆224Nov 16, 2024Updated last year
- All things awesome related to Dagster!☆148Jun 1, 2026Updated 3 months ago
- Personal Data Engineering Projects☆1,036Feb 8, 2023Updated 3 years ago
- Explore a curated collection of 100 practical JavaScript projects to sharpen your skills from beginner to advanced levels.☆327Aug 14, 2026Updated last month
- A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for …☆138Apr 18, 2020Updated 6 years ago
- DevOps Interview Questions and answers for juniors☆397May 6, 2024Updated 2 years ago
- Core Jupyter functionality☆216Sep 7, 2026Updated last week