Alternatives and similar repositories for pbspark:

Users that are interested in pbspark are comparing it to the libraries listed below

• delta-incubator / deltaray
  Delta reader for the Ray open-source toolkit for building ML applications
  ☆43Updated 11 months ago
• Nike-Inc / spark-expectations
  A Python Library to support running data quality rules while the spark job is running⚡
  ☆167Updated last week
• 1ambda / lakehouse
  Playground for Lakehouse (Iceberg, Hudi, Spark, Flink, Trino, DBT, Airflow, Kafka, Debezium CDC)
  ☆49Updated last year
• Wuerike / kafka-iceberg-streaming
  Docker envinroment to stream data from Kafka to Iceberg tables
  ☆24Updated 10 months ago
• gmyrianthous / dbt-airflow
  A Python package that creates fine-grained dbt tasks on Apache Airflow
  ☆62Updated 3 months ago
• mikulskibartosz / check-engine
  Data validation library for PySpark 3.0.0
  ☆34Updated 2 years ago
• skhatri / airflow-by-example
  Bunch of Airflow Configurations and DAGs for Kubernetes, Spark based data-pipelines. Scale inside Kubernetes using spark kubernetes maste…
  ☆22Updated 2 years ago
• jaceklaskowski / spark-kubernetes-book
  The Internals of Spark on Kubernetes
  ☆70Updated 2 years ago
• astronomer / astronomer-providers
  Airflow Providers containing Deferrable Operators & Sensors from Astronomer
  ☆142Updated this week
• dipankarmazumdar / awesome-lakehouse-guide
  Repo for everything open table formats (Iceberg, Hudi, Delta Lake) and the overall Lakehouse architecture
  ☆50Updated last week
• debussy-labs / debussy_concert
  Debussy is an opinionated Data Architecture and Engineering framework, enabling data analysts and engineers to build better platforms and…
  ☆28Updated last year
• dimajix / flowman
  Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pi…
  ☆93Updated this week
• cloudcheflabs / dataroaster
  ☆40Updated last year
• justhackit / spark-utils
  ☆10Updated 2 years ago
• getindata / doge-datagen
  ☆18Updated 2 years ago
• mrn-aglic / apache-iceberg-data-exploration
  ☆14Updated 11 months ago
• Nike-Inc / brickflow
  Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
  ☆192Updated last month
• acryldata / dbt-impact-action
  ☆14Updated 11 months ago
• feluelle / airflint
  Enforce Best Practices for all your Airflow DAGs. ⭐
  ☆93Updated this week
• tomasfarias / airflow-dbt-python
  A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.
  ☆184Updated 6 months ago
• open-metadata / openmetadata-helm-charts
  ☆49Updated last week
• Data-Engineering-Weekly / dataengineeringweekly
  Weekly Data Engineering Newsletter
  ☆94Updated 6 months ago
• garystafford / kafka-connect-msk-demo
  For a series of posts on Amazon MSK, Amazon EKS, and Amazon EMR
  ☆65Updated 3 years ago
• astronomer / airflow-provider-fivetran-async
  A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran
  ☆20Updated last week
• astronomer / cosmos-demo
  Demo DAGs that show how to run dbt Core in Airflow using Cosmos
  ☆52Updated 3 months ago
• developer-advocacy-dremio / definitive-guide-to-apache-iceberg
  ☆62Updated this week
• holdenk / spark-upgrade
  Magic to help Spark pipelines upgrade
  ☆34Updated 3 months ago
• criteo / cluster-pack
  A library on top of either pex or conda-pack to make your Python code easily available on a cluster
  ☆46Updated last month
• ronald-smith-angel / owl-data-sanitizer
  A pyspark lib to validate data quality
  ☆18Updated 2 years ago
• quby-io / databricks-workflow
  Example of a scalable IoT data processing pipeline setup using Databricks
  ☆31Updated 4 years ago