oap-project / pmem-shuffle
Spark* Shuffle plugin for support shuffling through remote persistent memory over fabrics, which leverages the RDMA network and remote persistent memory (for read) to provide extremely high performance and low latency shuffle solutions for Spark*.
☆14Updated last year
Alternatives and similar repositories for pmem-shuffle
Users that are interested in pmem-shuffle are comparing it to the libraries listed below
Sorting:
- This is archive of SparkRDMA project. The new repository with RDMA shuffle acceleration for Apache Spark is here: https://github.com/Nvid…☆246Updated 6 years ago
- TPC-H queries in Apache Spark SQL using native DataFrames API☆99Updated last year
- Mirror of Apache crail (Incubating)☆150Updated 2 years ago
- Naos: Serialization-free RDMA networking in Java☆15Updated 3 years ago
- Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange☆127Updated 4 months ago
- Gluten: Plugin to Boost Trino's Performance☆71Updated last year
- All the things about TPC-DS in Apache Spark☆105Updated last year
- Spark Shuffle Optimization with RDMA+AEP☆30Updated last year
- Use the TPC-DS benchmark to test Spark SQL performance☆179Updated 5 years ago
- Java bindings for https://github.com/facebookincubator/velox☆25Updated this week
- Spark* plug-in for accelerating Spark* SQL performance by using cache and index at SQL data source layer.☆37Updated 2 years ago
- A modular acceleration toolkit for big data analytic engines☆68Updated last year
- Lakehouse storage system benchmark☆73Updated 2 years ago
- Native SQL Engine plugin for Spark SQL with vectorized SIMD optimizations.☆257Updated 2 years ago
- A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer☆50Updated last year
- Performance Analysis Tool☆76Updated 2 years ago
- A hadoop compatible FUSE use for all.☆29Updated 7 months ago
- Star Schema Benchmark dbgen☆123Updated last year
- Trisk on Flink☆16Updated 2 years ago
- Spark Terasort☆122Updated 2 years ago
- DS2 is an auto-scaling controller for distributed streaming dataflows☆89Updated 2 years ago
- Spark* shuffle plugin for support shuffling data through a remote Hadoop-compatible file system, as opposed to vanilla Spark's local-dis…☆21Updated last year
- High Performance Network Library for RDMA☆27Updated 2 years ago
- Snowflake dataset containing statistics for 70 million queries over 14 day period☆112Updated 3 years ago
- DiSNI: Direct Storage and Networking Interface☆191Updated 2 years ago
- Fast, predictable data analytics based on (and API-compatible with) Apache Spark☆25Updated 7 years ago
- Smart Storage Management for Big Data, a comprehensive hot/cold data optimized solution☆142Updated 2 years ago
- An Extensible Data Skipping Framework☆46Updated 3 months ago
- The preview version of a spillable state backend for Apache Flink☆39Updated 4 years ago
- TPC-DS queries☆61Updated 9 years ago