crflynn / pbspark

Related projects ⓘ

Alternatives and complementary repositories for pbspark

• delta-incubator / deltaray
  Delta reader for the Ray open-source toolkit for building ML applications
  ☆42Updated 9 months ago
• mrpowers-io / jodie
  Delta lake and filesystem helper methods
  ☆49Updated 8 months ago
• mikulskibartosz / check-engine
  Data validation library for PySpark 3.0.0
  ☆34Updated 2 years ago
• SaurabhChawla100 / spark-radiant
  Spark-Radiant is Apache Spark Performance and Cost Optimizer
  ☆25Updated 2 years ago
• HeartSaVioR / spark-state-tools
  Spark Structured Streaming State Tools
  ☆34Updated 4 years ago
• jaceklaskowski / spark-kubernetes-book
  The Internals of Spark on Kubernetes
  ☆70Updated 2 years ago
• Nike-Inc / spark-expectations
  A Python Library to support running data quality rules while the spark job is running⚡
  ☆162Updated this week
• dipankarmazumdar / awesome-lakehouse-guide
  Repo for everything open table formats (Iceberg, Hudi, Delta Lake) and the overall Lakehouse architecture
  ☆28Updated 2 weeks ago
• justhackit / spark-utils
  ☆10Updated 2 years ago
• SemyonSinchenko / flake8-pyspark-with-column
  A flake8 plugin that detects of usage withColumn in a loop or inside reduce
  ☆21Updated last month
• rafaelpierre / pyjaws
  PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows
  ☆41Updated 4 months ago
• cloudcheflabs / dataroaster
  ☆40Updated last year
• getindata / doge-datagen
  ☆16Updated last year
• zheyuan28 / SparkTaskMetrics
  Task Metrics Explorer
  ☆13Updated 5 years ago
• yaooqinn / itachi
  A library that brings useful functions from various modern database management systems to Apache Spark
  ☆56Updated last year
• rajagurunath / lakehouse-sharing
  A Table format agnostic data sharing framework
  ☆38Updated 9 months ago
• open-metadata / openmetadata-helm-charts
  ☆44Updated last week
• target / data-validator
  A tool to validate data, built around Apache Spark.
  ☆101Updated this week
• delta-incubator / dat
  Delta Acceptance Testing
  ☆18Updated 3 months ago
• G-Research / spark-extension
  A library that provides useful extensions to Apache Spark and PySpark.
  ☆196Updated last week
• holdenk / spark-upgrade
  Magic to help Spark pipelines upgrade
  ☆33Updated last month
• mrpowers-io / levi
  Delta Lake helper methods. No Spark dependency.
  ☆22Updated 2 months ago
• dataflint / spark
  Performance Observability for Apache Spark
  ☆196Updated last week
• 1ambda / lakehouse
  Playground for Lakehouse (Iceberg, Hudi, Spark, Flink, Trino, DBT, Airflow, Kafka, Debezium CDC)
  ☆43Updated last year
• getindata / dbt-airflow-factory
  Library to convert DBT manifest metadata to Airflow tasks
  ☆46Updated 8 months ago
• AbsaOSS / hyperdrive
  Extensible streaming ingestion pipeline on top of Apache Spark
  ☆44Updated 7 months ago
• snowflakedb / spark-snowflake
  Snowflake Data Source for Apache Spark.
  ☆217Updated this week
• Nike-Inc / brickflow
  Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
  ☆187Updated last week
• SETL-Framework / setl
  A simple Spark-powered ETL framework that just works 🍺
  ☆178Updated 11 months ago