le-oasis / docker-airflow-spark

Related projects: ⓘ

• cordon-thiago / airflow-spark
  Docker with Airflow and Spark standalone cluster
  ☆239Updated last year
• josephmachado / data_engineering_project_template
  A template repository to create a data project with IAC, CI/CD, Data migrations, & testing
  ☆225Updated 2 months ago
• josephmachado / beginner_de_project_stream
  Simple stream processing pipeline
  ☆89Updated 3 months ago
• josephmachado / online_store
  End to end data engineering project
  ☆49Updated last year
• abdkumar / spotify-stream-analytics
  Generate synthetic Spotify music stream dataset to create dashboards. Spotify API generates fake event data emitted to Kafka. Spark consu…
  ☆66Updated 9 months ago
• alanchn31 / Movalytics-Data-Warehouse
  Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
  ☆127Updated 4 years ago
• josephmachado / data_engineering_best_practices
  Sample project to demonstrate data engineering best practices
  ☆156Updated 6 months ago
• dominikhei / Local-Data-LakeHouse
  Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testin…
  ☆51Updated last year
• arezamoosavi / AcidOnSpark-ETL
  Delta-Lake, ETL, Spark, Airflow
  ☆42Updated last year
• josephmachado / simple_dbt_project
  Code for dbt tutorial
  ☆138Updated 3 months ago
• Snowflake-Labs / sfguide-data-engineering-with-snowpark-python
  ☆102Updated last month
• lucaspfigueiredo / elt-pipeline
  Apartments Data Pipeline using Airflow and Spark.
  ☆18Updated 2 years ago
• itversity / data-engineering-spark
  ☆84Updated 2 years ago
• josephmachado / bitcoinMonitor
  Near real time ETL to populate a dashboard.
  ☆69Updated 3 months ago
• danielbeach / dataEngineeringTemplate
  Template for Data Engineering and Data Pipeline projects
  ☆101Updated last year
• SebasMBK / SunglassesHubETL
  This project is for demonstrating knowledge of Data Engineering tools and concepts and also learning in the process
  ☆44Updated last year
• jacob1421 / RustCheatersDataPipeline
  Data pipeline that scrapes Rust cheater Steam profiles
  ☆50Updated 2 years ago
• josephmachado / spark_submit_airflow
  Simple repo to demonstrate how to submit a spark job to EMR from Airflow
  ☆31Updated 3 years ago
• simardeep1792 / Data-Engineering-Streaming-Project
  ☆35Updated 2 months ago
• ajupton / big-data-engineering-project
  Big Data Engineering practice project, including ETL with Airflow and Spark using AWS S3 and EMR
  ☆79Updated 5 years ago
• afaqueahmad7117 / spark-experiments
  Ultimate guide for mastering Spark Performance Tuning and Optimization concepts and for preparing for Data Engineering interviews
  ☆57Updated 4 months ago
• josephmachado / analytical_dp_with_sql
  Code for my "Efficient Data Processing in SQL" book.
  ☆47Updated last month
• iamraphson / DE-2024-project-book-recommendation
  ☆22Updated 5 months ago
• andrem8 / surf_dash
  ☆124Updated 2 years ago
• pyjaime / docker-airflow-spark
  Docker with Airflow + Postgres + Spark cluster + JDK (spark-submit support) + Jupyter Notebooks
  ☆20Updated 2 years ago
• konosp / dbt-airflow-docker-compose
  Execution of DBT models using Apache Airflow through Docker Compose
  ☆111Updated last year
• bruno-szdl / cheatsheets
  Repo for saving cheat sheets
  ☆42Updated 3 months ago
• josephmachado / python_essentials_for_data_engineers
  Code for blog at https://www.startdataengineering.com/post/python-for-de/
  ☆47Updated 3 months ago
• gmyrianthous / dbt-airflow
  A Python package that creates fine-grained dbt tasks on Apache Airflow
  ☆52Updated 5 months ago
• iobruno / data-engineering-zoomcamp
  Data Engineering examples covering Airflow and Mage for workflows; dbt for BigQuery, Redshift, ClickHouse; Spark and Kafka for Batch/Stre…
  ☆50Updated 3 weeks ago