okzapradhana / etl-flatfile-airflow
Building Data Warehouse on BigQuery which takes flat file as the data sources with Airflow as the Orchestrator
☆12Updated 3 years ago
Alternatives and similar repositories for etl-flatfile-airflow:
Users that are interested in etl-flatfile-airflow are comparing it to the libraries listed below
- End-to-end ELT data engineering project☆21Updated 2 years ago
- Data lake, data warehouse on GCP☆56Updated 3 years ago
- Project for "Data pipeline design patterns" blog.☆45Updated 8 months ago
- Simple ETL pipeline using Python☆26Updated last year
- For this project I am creating an ETL (Extract, Transform, and Load) pipeline using Python, RegEx, and SQL Database. The goal is to retri…☆27Updated 4 years ago
- ☆11Updated 3 years ago
- DataTalks.Club's Data Engineering Zoomcamp Project☆23Updated 2 years ago
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆35Updated last year
- ☆35Updated 4 months ago
- Schedule a data pipeline in Google Cloud using cloud function, BigQuery, cloud storage, cloud scheduler, stack trace, cloud build, and p…☆26Updated 5 years ago
- Demonstrating the concept of Google PubSub, a messaging queue service in Google, thru streaming fake financial data thru PubSub and query…☆15Updated 7 years ago
- Stream Avro SpecificRecord objects in BigQuery using Cloud Dataflow☆13Updated 3 years ago
- This is a simple ETL project with Python :)☆29Updated 2 years ago
- This repository contains an example of how to leverage Cloud Composer and Cloud Dataflow to move data from a Microsoft SQL Server to BigQ…☆18Updated 11 months ago
- Data Engineering with Google Cloud Platform, published by Packt☆117Updated last year
- Spark, Airflow, Kafka☆26Updated last year
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆16Updated 5 years ago
- Classwork projects and home works done through Udacity data engineering nano degree☆74Updated last year
- A self-contained, ready to run Airflow and Kafka project. Can be run locally or within codespaces.☆17Updated last year
- Data Engineering on GCP☆35Updated 2 years ago
- Code base for airflow training series Getting easy with Apache Airflow☆39Updated last year
- ☆13Updated 9 months ago
- ☆16Updated last year
- A data engineering project with Airflow, dbt, Terrafrom, GCP and much more!☆24Updated 2 years ago
- ☆38Updated 2 years ago
- Creation of a data lakehouse and an ELT pipeline to enable the efficient analysis and use of data☆45Updated last year
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆27Updated last year
- learning-by-doing data model built with dbt-core☆12Updated 4 months ago
- 😈Complete End to End ETL Pipeline with Spark, Airflow, & AWS☆45Updated 5 years ago
- Python ETL demo for Hackforge☆31Updated last year