paypay / DataEngineerChallenge

Related projects ⓘ

Alternatives and complementary repositories for DataEngineerChallenge

• Data-Engineering-Weekly / dataengineeringweekly
  Weekly Data Engineering Newsletter
  ☆93Updated 4 months ago
• yennanliu / spark-etl-pipeline
  Various data stream/batch process demo with Apache Scala Spark 🚀
  ☆11Updated 4 years ago
• siddd88 / gcp-data-engineering
  Batch Processing , orchestration using Apache Airflow and Google Workflows, spark structured Streaming and a lot more
  ☆19Updated 2 years ago
• MrPowers / gill
  An example PySpark project with pytest
  ☆17Updated 7 years ago
• spotify / scio.g8
  A Giter8 template for scio
  ☆30Updated 3 weeks ago
• deliveryhero / pyconde2019-airflow-ml-workshop
  PyConDE & PyData Berlin 2019 Airflow Workshop: Airflow for machine learning pipelines.
  ☆46Updated last year
• holdenk / spark-upgrade
  Magic to help Spark pipelines upgrade
  ☆33Updated last month
• marcosmarxm / airflow-testing-ci-workflow
  (project & tutorial) dag pipeline tests + ci/cd setup
  ☆85Updated 3 years ago
• yennanliu / DE-100-days
  data engineering 100 days 🤖 🧲 🦾 | #DE
  ☆37Updated last year
• holdenk / spark-flowchart
  Flowchart for debugging Spark applications
  ☆101Updated last month
• mikulskibartosz / check-engine
  Data validation library for PySpark 3.0.0
  ☆34Updated 2 years ago
• anish749 / spark2-etl-examples
  A project with examples of using few commonly used data manipulation/processing/transformation APIs in Apache Spark 2.0.0
  ☆25Updated 3 years ago
• karenbajador / pyspark_greatexpectations
  ☆11Updated 2 years ago
• ananthdurai / airflow-training
  Airflow training for the crunch conf
  ☆105Updated 6 years ago
• bosea / spark-unit-testing
  A tutorial on Apache Spark Unit Testing
  ☆37Updated 8 years ago
• holdenk / sparkProjectTemplate.g8
  Template for Spark Projects
  ☆101Updated 5 months ago
• fastforwardlabs / cml_churn_demo_mlops
  ☆16Updated last year
• Wittline / pyspark-on-aws-emr
  The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on…
  ☆25Updated 2 years ago
• rockthejvm / spark-performance-tuning
  The official repository for the Rock the JVM Spark Optimization 2 course
  ☆37Updated 11 months ago
• godatadriven / airflow-training-skeleton
  Skeleton project for Apache Airflow training participants to work on.
  ☆16Updated 4 years ago
• vivek-bombatkar / MyLearningNotes
  Because its never late to start taking notes and 'public' it...
  ☆60Updated 3 weeks ago
• godatadriven / iterative-broadcast-join
  The iterative broadcast join example code.
  ☆69Updated 7 years ago
• NashTech-Labs / Lambda-Arch-Spark
  ☆74Updated 4 years ago
• SaurabhChawla100 / spark-radiant
  Spark-Radiant is Apache Spark Performance and Cost Optimizer
  ☆25Updated 2 years ago
• PacktPublishing / Building-Big-Data-Pipelines-with-Apache-Beam
  Building Big Data Pipelines with Apache Beam, published by Packt
  ☆83Updated last year
• vbounyasit / MyDataFramework
  An ETL framework in Scala for Data Engineers
  ☆22Updated 2 years ago
• MrPowers / bebe
  Filling in the Spark function gaps across APIs
  ☆50Updated 3 years ago
• hjhuney / An-Intro-to-Scala
  An Introduction to Scala
  ☆23Updated last year
• awesome-spark / learn-by-examples
  Real-world Spark pipelines examples
  ☆83Updated 6 years ago
• moyano83 / High-Performance-Spark
  ☆30Updated 5 years ago