okzapradhana / etl-flatfile-airflow
Building Data Warehouse on BigQuery which takes flat file as the data sources with Airflow as the Orchestrator
☆11Updated 3 years ago
Related projects ⓘ
Alternatives and complementary repositories for etl-flatfile-airflow
- Data lake, data warehouse on GCP☆54Updated 2 years ago
- End-to-end ELT data engineering project☆20Updated last year
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆37Updated last year
- A data engineering project with Airflow, dbt, Terrafrom, GCP and much more!☆22Updated 2 years ago
- Simple ETL pipeline using Python☆21Updated last year
- Data Engineering with Google Cloud Platform, published by Packt☆109Updated last year
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆35Updated last year
- 😈Complete End to End ETL Pipeline with Spark, Airflow, & AWS☆43Updated 5 years ago
- ☆38Updated 4 months ago
- Python and AirFlow - Data Pipeline Orchestration☆16Updated last year
- Tutorial for easy-to-manage data pipelines with Airflow☆9Updated 2 years ago
- ☆11Updated 2 years ago
- This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging…☆74Updated 5 years ago
- DataTalks.Club's Data Engineering Zoomcamp Project☆22Updated 2 years ago
- ☆86Updated 2 years ago
- Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO☆57Updated last year
- Big Data Engineering practice project, including ETL with Airflow and Spark using AWS S3 and EMR☆80Updated 5 years ago
- Schedule a data pipeline in Google Cloud using cloud function, BigQuery, cloud storage, cloud scheduler, stack trace, cloud build, and p…☆26Updated 5 years ago
- Cloud Functions streaming insert to BigQuery (with Cloud Pub/Sub trigger). In this example, the function will make a REST API call to get…☆26Updated last year
- ☆27Updated last year
- Data Pipeline from the Global Historical Climatology Network DataSet☆24Updated last year
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆11Updated last year
- Demonstration of using Apache Spark to build robust ETL pipelines while taking advantage of open source, general purpose cluster computin…☆24Updated last year
- This is a simple ETL project with Python :)☆26Updated 2 years ago
- Data Engineering on GCP☆30Updated 2 years ago
- Classwork projects and home works done through Udacity data engineering nano degree☆74Updated 11 months ago
- Courses and projects on Data Camp☆11Updated 4 years ago
- An end-to-end data engineering pipeline to create a dashboard for the latest content on the r/Stocks subreddit☆19Updated 2 years ago
- ☆36Updated last year
- Spark, Airflow, Kafka☆26Updated last year