ayyoubmaul / hadoop-docker

Related projects ⓘ

Alternatives and complementary repositories for hadoop-docker

• cordon-thiago / airflow-spark
  Docker with Airflow and Spark standalone cluster
  ☆244Updated last year
• josephmachado / spark_submit_airflow
  Simple repo to demonstrate how to submit a spark job to EMR from Airflow
  ☆32Updated 4 years ago
• josephmachado / data_engineering_project_template
  A template repository to create a data project with IAC, CI/CD, Data migrations, & testing
  ☆237Updated 4 months ago
• josephmachado / beginner_de_project_stream
  Simple stream processing pipeline
  ☆91Updated 4 months ago
• arezamoosavi / AcidOnSpark-ETL
  Delta-Lake, ETL, Spark, Airflow
  ☆44Updated 2 years ago
• josephmachado / data_engineering_best_practices
  Sample project to demonstrate data engineering best practices
  ☆164Updated 8 months ago
• josephmachado / online_store
  End to end data engineering project
  ☆49Updated 2 years ago
• cartershanklin / pyspark-cheatsheet
  PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
  ☆422Updated 3 weeks ago
• dogukannulu / kafka_spark_structured_streaming
  Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra
  ☆128Updated last year
• ovimihai / airflow-cert-dag-authoring
  Course notes for the Astronomer Certification DAG Authoring for Apache Airflow
  ☆47Updated 7 months ago
• subhamkharwal / ease-with-apache-spark
  Series follows learning from Apache Spark (PySpark) with quick tips and workaround for daily problems in hand
  ☆42Updated last year
• josephmachado / simple_dbt_project
  Code for dbt tutorial
  ☆143Updated 5 months ago
• josephmachado / socialetl
  Project for "Data pipeline design patterns" blog.
  ☆41Updated 3 months ago
• alanchn31 / Movalytics-Data-Warehouse
  Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
  ☆133Updated 4 years ago
• astronomer / airflow-data-quality-demo
  A repository of sample code to show data quality checking best practices using Airflow.
  ☆72Updated last year
• gmyrianthous / dbt-airflow
  A Python package that creates fine-grained dbt tasks on Apache Airflow
  ☆62Updated last month
• iobruno / data-engineering-zoomcamp
  Data Engineering examples for Airflow, Prefect, and Mage.ai; dbt for BigQuery, Redshift, ClickHouse, PostgreSQL; Spark/PySpark for Batch …
  ☆50Updated this week
• dogukannulu / streaming_data_processing
  Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO
  ☆57Updated last year
• lucaspfigueiredo / elt-pipeline
  Apartments Data Pipeline using Airflow and Spark.
  ☆18Updated 2 years ago
• Marcel-Jan / docker-hadoop-spark
  Multi-container environment with Hadoop, Spark and Hive
  ☆202Updated 10 months ago
• kaoutaar / end-to-end-etl-pipeline-jcdecaux-API
  velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docke…
  ☆18Updated 2 months ago
• MrPowers / mack
  Delta Lake helper methods in PySpark
  ☆304Updated 2 months ago
• 1ambda / lakehouse
  Playground for Lakehouse (Iceberg, Hudi, Spark, Flink, Trino, DBT, Airflow, Kafka, Debezium CDC)
  ☆43Updated last year
• ananthdurai / airflow-training
  Airflow training for the crunch conf
  ☆105Updated 6 years ago
• developershomes / SparkETL
  Spark all the ETL Pipelines
  ☆32Updated last year
• hyunjoonbok / PySpark
  PySpark functions and utilities with examples. Assists ETL process of data modeling
  ☆99Updated 3 years ago
• yTek01 / docker-spark-airflow
  ☆31Updated last year
• derar-alhussein / Databricks-Certified-Data-Engineer-Professional
  The resources of the preparation course for Databricks Data Engineer Professional certification exam
  ☆84Updated 3 weeks ago
• abdkumar / spotify-stream-analytics
  Generate synthetic Spotify music stream dataset to create dashboards. Spotify API generates fake event data emitted to Kafka. Spark consu…
  ☆66Updated 10 months ago
• xnuinside / airflow_in_docker_compose
  Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)
  ☆184Updated 11 months ago