Stefen-Taime / ETL-Data-Pipeline-RDBMS-TO-HDFS-using-Airflow-Apache-Sqoop-Spark-Postgres-and-Hive

Related projects ⓘ

Alternatives and complementary repositories for ETL-Data-Pipeline-RDBMS-TO-HDFS-using-Airflow-Apache-Sqoop-Spark-Postgres-and-Hive

• Pathairush / rdbms_to_hdfs_data_pipeline
  A data pipeline moving data from a Relational database system (RDBMS) to a Hadoop file system (HDFS).
  ☆15Updated 3 years ago
• cordon-thiago / airflow-spark
  Docker with Airflow and Spark standalone cluster
  ☆246Updated last year
• alanchn31 / Movalytics-Data-Warehouse
  Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
  ☆133Updated 4 years ago
• josephmachado / online_store
  End to end data engineering project
  ☆51Updated 2 years ago
• cartershanklin / pyspark-cheatsheet
  PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
  ☆424Updated last month
• josephmachado / spark_submit_airflow
  Simple repo to demonstrate how to submit a spark job to EMR from Airflow
  ☆32Updated 4 years ago
• josephmachado / data_engineering_project_template
  A template repository to create a data project with IAC, CI/CD, Data migrations, & testing
  ☆241Updated 4 months ago
• data-engineering-community / data-engineering-project-template
  This is a template you can use for your next data engineering portfolio project.
  ☆163Updated 3 years ago
• josephmachado / bitcoinMonitor
  Near real time ETL to populate a dashboard.
  ☆70Updated 5 months ago
• ris-tlp / audiophile-e2e-pipeline
  Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.
  ☆197Updated last year
• RSKriegs / finnhub-streaming-data-pipeline
  Stream processing pipeline from Finnhub websocket using Spark, Kafka, Kubernetes and more
  ☆294Updated 11 months ago
• josephmachado / data_engineering_best_practices
  Sample project to demonstrate data engineering best practices
  ☆166Updated 8 months ago
• andrem8 / surf_dash
  ☆132Updated 2 years ago
• supratim94336 / DataEngineeringCapstoneProject
  😈Complete End to End ETL Pipeline with Spark, Airflow, & AWS
  ☆43Updated 5 years ago
• ankurchavda / streamify
  A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
  ☆575Updated 2 years ago
• le-oasis / docker-airflow-spark
  Building a Modern Data Lake with Minio, Spark, Airflow via Docker.
  ☆15Updated 6 months ago
• ABZ-Aaron / Reddit-API-Pipeline
  ☆310Updated 2 months ago
• josephmachado / beginner_de_project_stream
  Simple stream processing pipeline
  ☆92Updated 5 months ago
• Snowflake-Labs / sfguide-data-engineering-with-snowpark-python
  ☆113Updated last month
• ajupton / big-data-engineering-project
  Big Data Engineering practice project, including ETL with Airflow and Spark using AWS S3 and EMR
  ☆80Updated 5 years ago
• sidharth1805 / Spotify_etl
  ☆128Updated last year
• arezamoosavi / AcidOnSpark-ETL
  Delta-Lake, ETL, Spark, Airflow
  ☆44Updated 2 years ago
• tatwan / airflow-spark-aws-emr
  ☆11Updated 3 years ago
• hyunjoonbok / PySpark
  PySpark functions and utilities with examples. Assists ETL process of data modeling
  ☆99Updated 3 years ago
• afaqueahmad7117 / spark-experiments
  Ultimate guide for mastering Spark Performance Tuning and Optimization concepts and for preparing for Data Engineering interviews
  ☆70Updated 6 months ago
• LearningJournal / Spark-Streaming-In-Python
  Apache Spark 3 - Structured Streaming Course Material
  ☆119Updated last year
• josephmachado / simple_dbt_project
  Code for dbt tutorial
  ☆143Updated 5 months ago
• josephmachado / efficient_data_processing_spark
  Code for "Efficient Data Processing in Spark" Course
  ☆247Updated last month
• vivek-bombatkar / Spark-with-Python---My-learning-notes-
  ETL pipeline using pyspark (Spark - Python)
  ☆108Updated 4 years ago