Alternatives and similar repositories for aiperf

Users that are interested in aiperf are comparing it to the libraries listed below

• sgl-project / genai-bench
  Genai-bench is a powerful benchmark tool designed for comprehensive token-level performance evaluation of large language model (LLM) serv…
  ☆251Updated this week
• ai-dynamo / nixl
  NVIDIA Inference Xfer Library (NIXL)
  ☆788Updated this week
• HabanaAI / vllm-fork
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆85Updated this week
• coreweave / nccl-tests
  NVIDIA NCCL Tests for Distributed Training
  ☆132Updated this week
• sgl-project / ome
  Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, T…
  ☆355Updated this week
• microsoft / sarathi-serve
  A low-latency & high-throughput serving engine for LLMs
  ☆462Updated 2 months ago
• snowflakedb / ArcticInference
  ArcticInference: vLLM plugin for high-throughput, low-latency inference
  ☆363Updated last week
• ai-dynamo / aiconfigurator
  Offline optimization of your disaggregated Dynamo graph
  ☆137Updated this week
• NVIDIA / nvidia-resiliency-ext
  NVIDIA Resiliency Extension is a python package for framework developers and users to implement fault-tolerant features. It improves the …
  ☆243Updated 2 weeks ago
• microsoft / vattention
  Dynamic Memory Management for Serving LLMs without PagedAttention
  ☆454Updated 7 months ago
• perplexityai / pplx-kernels
  Perplexity GPU Kernels
  ☆548Updated 2 months ago
• triton-inference-server / perf_analyzer
  ☆131Updated 3 weeks ago
• AlibabaPAI / llumnix
  Efficient and easy multi-instance LLM serving
  ☆520Updated 4 months ago
• meta-pytorch / torchcomms
  torchcomms: a modern PyTorch communications API
  ☆319Updated this week
• vllm-project / flash-attention
  Fast and memory-efficient exact attention
  ☆107Updated 3 weeks ago
• ovg-project / kvcached
  Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
  ☆735Updated last month
• vllm-project / router
  A high-performance and light-weight router for vLLM large scale deployment
  ☆80Updated 2 weeks ago
• bytedance / InfiniStore
  KV cache store for distributed LLM inference
  ☆384Updated last month
• NVIDIA / cuda-checkpoint
  CUDA checkpoint and restore utility
  ☆401Updated 3 months ago
• vllm-project / speculators
  A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM
  ☆182Updated this week
• vllm-project / compressed-tensors
  A safetensors extension to efficiently store sparse quantized tensors on disk
  ☆228Updated this week
• SqueezeBits / QUICK
  QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  ☆118Updated last year
• microsoft / mscclpp
  MSCCL++: A GPU-driven communication stack for scalable AI applications
  ☆449Updated this week
• neuralmagic / AutoFP8
  ☆206Updated 8 months ago
• Azure / msccl
  Microsoft Collective Communication Library
  ☆66Updated last year
• perplexityai / pplx-garden
  Perplexity open source garden for inference technology
  ☆324Updated 2 weeks ago
• vllm-project / tpu-inference
  TPU inference for vLLM, with unified JAX and PyTorch support.
  ☆207Updated this week
• cli99 / llm-analysis
  Latency and Memory Analysis of Transformer Models for Training and Inference
  ☆475Updated 8 months ago
• triton-inference-server / vllm_backend
  ☆322Updated this week
• meta-pytorch / applied-ai
  Applied AI experiments and examples for PyTorch
  ☆312Updated 4 months ago