Spark* Shuffle plugin for support shuffling through remote persistent memory over fabrics, which leverages the RDMA network and remote persistent memory (for read) to provide extremely high performance and low latency shuffle solutions for Spark*.
☆14Sep 18, 2023Updated 2 years ago
Alternatives and similar repositories for pmem-shuffle
Users that are interested in pmem-shuffle are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Spark* shuffle plugin for support shuffling data through a remote Hadoop-compatible file system, as opposed to vanilla Spark's local-dis…☆21Mar 15, 2024Updated 2 years ago
- Source code for XPGraph-MICRO22☆11Apr 10, 2023Updated 3 years ago
- Spark Shuffle Optimization with RDMA+AEP☆30May 23, 2023Updated 3 years ago
- Tools for building, packaging, and OAP public cloud integrations such as AWS EMR, Google Dataproc and K8S.☆18Mar 27, 2024Updated 2 years ago
- Spark* plug-in for accelerating Spark* SQL performance by using cache and index at SQL data source layer.☆37Jan 3, 2023Updated 3 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Hadoop InputFormat for http://druid.io/☆10Oct 26, 2016Updated 9 years ago
- A Python library and command line utility for manipulating and plotting stellar lightcurves.☆10Jun 14, 2016Updated 10 years ago
- ☆12Apr 7, 2025Updated last year
- R package implementing the Nearest Neighbor Descent method for approximate nearest neighbors☆17Updated this week
- 基于多线程与epoll的高并发TCP服务器☆11Aug 4, 2018Updated 8 years ago
- ☆12Mar 13, 2024Updated 2 years ago
- Alerting and monitoring tool for Apache Spark☆23May 20, 2022Updated 4 years ago
- High Performance Network Library for RDMA☆28Jan 3, 2023Updated 3 years ago
- HMM-guided metagenomic gene-targeted assembler using iterative de Bruijn graphs☆18Oct 3, 2016Updated 9 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Firestorm is a Remote Shuffle Service, and provides the capability for Apache Spark and Apache Hadoop MapReduce applications to store shu…☆256Apr 7, 2023Updated 3 years ago
- Apache Spark - A unified analytics engine for large-scale data processing☆15Jul 24, 2023Updated 3 years ago
- A Python interface to gb-io, a fast GenBank parser written in Rust.☆24May 21, 2026Updated 2 months ago
- Java implementation of the SCRAM SASL for both server and client plus examples☆17Apr 18, 2021Updated 5 years ago
- benchmark-for-spark☆19May 7, 2025Updated last year
- A S3 Shuffle plugin for Apache Spark to enable elastic scaling for generic Spark workloads.☆52Sep 17, 2025Updated 10 months ago
- Passive Disaggregated Persistent Memory at USENIX ATC 2020.☆52Nov 2, 2020Updated 5 years ago
- ☆18Dec 30, 2025Updated 7 months ago
- Text Preprocessing in Python☆19Jan 15, 2017Updated 9 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Remote Persistent Memory Access Library☆105Sep 5, 2023Updated 2 years ago
- ViennaRNA Package consists of a C code library for the prediction and comparison of RNA secondary structures☆15May 20, 2022Updated 4 years ago
- Python Repository of the Institute of Astronomy @ KU Leuven☆20Nov 5, 2020Updated 5 years ago
- Remote shuffle service for Apache Spark to store shuffle data on remote servers.☆335Sep 29, 2023Updated 2 years ago
- Mirror of Apache livy (Incubating)☆13Jul 7, 2026Updated last month
- Parquet file generator☆22Apr 17, 2018Updated 8 years ago
- Naos: Serialization-free RDMA networking in Java☆17Aug 17, 2021Updated 4 years ago
- Log driver plugin for docker explained. The boilerplate code here can also be used to write your own driver if you are feeling adventurou…☆13Mar 13, 2019Updated 7 years ago
- HDFS based on Java implementation as a remote ObjectStore for DataFusion☆10Feb 13, 2024Updated 2 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- Latest PASTE (NSDI'18) repository☆13May 2, 2022Updated 4 years ago
- Plugin to accelerate Spark SQL with the NEC Vector Engine.☆19Aug 15, 2022Updated 4 years ago
- Symbolic range analysis for LLVM.☆12Jan 10, 2016Updated 10 years ago
- template for https://cnli.me☆10Feb 27, 2025Updated last year
- On the fly, translation of Spark programs to run natively on your Oracle DB. Your Spark programs require no changes.☆36Apr 15, 2025Updated last year
- Symbolic Liveness Analysis of real-world software building upon KLEE to detect liveness violations (e.g. infinite loop bugs)☆12Dec 16, 2021Updated 4 years ago
- Client libraries of end users of Apache Kyuubi☆11May 15, 2026Updated 3 months ago