dogukannulu / csv_extract_airflow_docker
Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.
☆35Updated last year
Alternatives and similar repositories for csv_extract_airflow_docker:
Users that are interested in csv_extract_airflow_docker are comparing it to the libraries listed below
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆39Updated last year
- ☆41Updated 8 months ago
- ☆27Updated last year
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆136Updated last year
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆35Updated last year
- ☆87Updated 2 years ago
- End to end data engineering project☆53Updated 2 years ago
- Simple ETL pipeline using Python☆25Updated last year
- YouTube tutorial project☆100Updated last year
- Code for blog at https://www.startdataengineering.com/post/python-for-de/☆71Updated 9 months ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆85Updated 7 months ago
- ☆32Updated last year
- ☆16Updated 11 months ago
- Projects done in the Data Engineer Nanodegree Program by Udacity.com☆135Updated 2 years ago
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆27Updated last year
- ☆135Updated 2 years ago
- 😈Complete End to End ETL Pipeline with Spark, Airflow, & AWS☆44Updated 5 years ago
- ☆37Updated last year
- Near real time ETL to populate a dashboard.☆73Updated 9 months ago
- ☆151Updated 2 years ago
- PySpark functions and utilities with examples. Assists ETL process of data modeling☆100Updated 4 years ago
- Classwork projects and home works done through Udacity data engineering nano degree☆74Updated last year
- For this project I am creating an ETL (Extract, Transform, and Load) pipeline using Python, RegEx, and SQL Database. The goal is to retri…☆27Updated 4 years ago
- Data Pipeline from the Global Historical Climatology Network DataSet☆27Updated 2 years ago
- Realtime Data Engineering Project☆26Updated 2 months ago
- PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like…☆101Updated last year
- ☆16Updated last year
- A data engineering project with Airflow, dbt, Terrafrom, GCP and much more!☆23Updated 2 years ago
- Data Engineering with Google Cloud Platform, published by Packt☆113Updated last year
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆65Updated last year