Alternatives and similar repositories for airflow-spark-minio-postgres

Users that are interested in airflow-spark-minio-postgres are comparing it to the libraries listed below

• dogukannulu / csv_extract_airflow_docker
  Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.
  ☆35Updated last year
• le-oasis / docker-airflow-spark
  Building a Modern Data Lake with Minio, Spark, Airflow via Docker.
  ☆20Updated last year
• dsynkov / spark-livy-on-airflow-workspace
  A workspace to experiment with Apache Spark, Livy, and Airflow in a Docker environment.
  ☆38Updated 4 years ago
• AnandDedha / dbt-airflow-demo
  A Python package that creates fine-grained dbt tasks on Apache Airflow
  ☆12Updated last year
• Armaan1Gohil / dataengineering-tech-stack
  Local Environment to Practice Data Engineering
  ☆142Updated 4 months ago
• yTek01 / docker-spark-airflow
  ☆37Updated 2 years ago
• PacktPublishing / Bigdata-on-Kubernetes
  Bigdata on Kubernetes, Published by Packt
  ☆31Updated 7 months ago
• alonsomedo / os-data-stack
  Building a Data Pipeline with an Open Source Stack
  ☆54Updated 10 months ago
• damklis / etljob
  Simple ETL pipeline using Python
  ☆26Updated last year
• ongxuanhong / de02-pyspark-optimization
  ☆14Updated 2 years ago
• arezamoosavi / AcidOnSpark-ETL
  Delta-Lake, ETL, Spark, Airflow
  ☆47Updated 2 years ago
• dogukannulu / airflow_kafka_cassandra_mongodb
  Produce Kafka messages, consume them and upload into Cassandra, MongoDB.
  ☆42Updated last year
• dogukannulu / streaming_data_processing
  Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO
  ☆60Updated last year
• ddgope / Data-Pipelines-with-Airflow
  This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging…
  ☆86Updated 5 years ago
• pyjaime / docker-airflow-spark
  Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks
  ☆23Updated 3 years ago
• itversity / data-engineering-spark
  ☆87Updated 2 years ago
• josephmachado / online_store
  End to end data engineering project
  ☆54Updated 2 years ago
• cordon-thiago / airflow-spark
  Docker with Airflow and Spark standalone cluster
  ☆257Updated last year
• konosp / dbt-airflow-docker-compose
  Execution of DBT models using Apache Airflow through Docker Compose
  ☆116Updated 2 years ago
• priye-1 / airflow_data_pipeline
  ☆16Updated last year
• muhammetbektas / spark_clickhouse_streaming
  Realtime Data Engineering Project
  ☆30Updated 4 months ago
• astronautyates / AirflowSnowflakeDBTQuickstart
  ☆22Updated last year
• mrn-aglic / spark-standalone-cluster
  This repo contains a spark standalone cluster on docker for anyone who wants to play with PySpark by submitting their applications.
  ☆34Updated last year
• josephmachado / bitcoinMonitor
  Near real time ETL to populate a dashboard.
  ☆72Updated 11 months ago
• simardeep1792 / Data-Engineering-Streaming-Project
  ☆40Updated 10 months ago
• airscholar / SparkingFlow
  This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python…
  ☆42Updated last year
• ajupton / big-data-engineering-project
  Big Data Engineering practice project, including ETL with Airflow and Spark using AWS S3 and EMR
  ☆83Updated 5 years ago
• TJaniF / airflow-elt-blueprint
  A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.
  ☆68Updated last year
• zkan / dtc-data-engineering-zoomcamp-project
  DataTalks.Club's Data Engineering Zoomcamp Project
  ☆23Updated 2 years ago
• guoliveira / data-engineer-zoomcamp-project
  ☆38Updated 2 years ago