Alternatives and similar repositories for pyspark

Users that are interested in pyspark are comparing it to the libraries listed below

• malexer / pytest-spark
  pytest plugin to run the tests with support of pyspark
  ☆88Updated 8 months ago
• mikulskibartosz / check-engine
  Data validation library for PySpark 3.0.0
  ☆33Updated 3 years ago
• BasPH / pylint-airflow
  Pylint plugin for static code analysis on Airflow code
  ☆97Updated 5 years ago
• G-Research / spark-extension
  A library that provides useful extensions to Apache Spark and PySpark.
  ☆232Updated last week
• itsjafer / jupyterlab-sparkmonitor
  JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook
  ☆92Updated 3 years ago
• datamechanics / delight
  A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.
  ☆346Updated last year
• cloudera / dbt-spark-livy
  The dbt-spark-livy adapter allows you to use dbt along with Apache Spark, by connecting via Apache Livy
  ☆12Updated 2 years ago
• dsaidgovsg / airflow-pipeline
  An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR
  ☆175Updated 8 months ago
• criteo / cluster-pack
  A library on top of either pex or conda-pack to make your Python code easily available on a cluster
  ☆46Updated last week
• getindata / dbt-airflow-factory
  Library to convert DBT manifest metadata to Airflow tasks
  ☆49Updated last month
• zero323 / pyspark-stubs
  Apache (Py)Spark type annotations (stub files).
  ☆118Updated 3 years ago
• mrpowers-io / spark-style-guide
  Spark style guide
  ☆272Updated last year
• victorcouste / trino-dbt-demo
  Trino dbt demo project to mix and load BigQuery data with and in a local PostgreSQL database
  ☆76Updated 4 years ago
• castor-team / airflow-castor
  ☆42Updated 4 years ago
• margitaii / pydeequ
  Python API for Deequ
  ☆41Updated 5 years ago
• godatadriven / whirl
  Fast iterative local development and testing of Apache Airflow workflows
  ☆203Updated last month
• arempter / hive-metastore-docker
  Example for article Running Spark 3 with standalone Hive Metastore 3.0
  ☆103Updated 3 years ago
• SETL-Framework / setl
  A simple Spark-powered ETL framework that just works 🍺
  ☆182Updated 4 months ago
• danielvdende / data-testing-with-airflow
  ☆201Updated 2 years ago
• JahstreetOrg / spark-on-kubernetes-helm
  Spark on Kubernetes infrastructure Helm charts repo
  ☆203Updated 3 years ago
• yaooqinn / itachi
  A library that brings useful functions from various modern database management systems to Apache Spark
  ☆61Updated 2 years ago
• julioasotodv / spark-df-profiling
  Create HTML profiling reports from Apache Spark DataFrames
  ☆197Updated 6 years ago
• astronomer / ap-airflow
  Astronomer Core Docker Images
  ☆105Updated last year
• sodadata / soda-spark
  Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes
  ☆63Updated 3 years ago
• jaceklaskowski / spark-kubernetes-book
  The Internals of Spark on Kubernetes
  ☆72Updated 3 years ago
• amundsen-io / amundsendatabuilder
  Data ingestion library for Amundsen to build graph and search index
  ☆204Updated last year
• zero323 / pyspark-asyncactions
  Asynchronous actions for PySpark
  ☆48Updated 4 years ago
• Nike-Inc / spark-expectations
  A Python Library to support running data quality rules while the spark job is running⚡
  ☆194Updated this week
• cerndb / spark-dashboard
  Spark-Dashboard is a solution for monitoring Apache Spark jobs. This repository provides the tooling and configuration for deploying an A…
  ☆132Updated 3 weeks ago
• feluelle / airflint
  Enforce Best Practices for all your Airflow DAGs. ⭐
  ☆108Updated last week