Alternatives and similar repositories for LLM-Inference-Bench

Users that are interested in LLM-Inference-Bench are comparing it to the libraries listed below

• microsoft / tokenweave
  Accepted to MLSys 2026
  ☆70Updated last week
• tyler-griggs / melange-release
  ☆48Updated last year
• project-etalon / etalon
  LLM Serving Performance Evaluation Harness
  ☆83Updated 11 months ago
• hao-ai-lab / MuxServe
  ☆84Updated 3 months ago
• AlibabaResearch / flash-llm
  Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity
  ☆233Updated 2 years ago
• DeepLink-org / DLSlime
  DLSlime: Flexible & Efficient Heterogeneous Transfer Toolkit
  ☆92Updated last week
• flexflow / flexflow-serve
  FlexFlow Serve: Low-Latency, High-Performance LLM Serving
  ☆71Updated 4 months ago
• PKU-SEC-Lab / HybriMoE
  [DAC'25] Official implement of "HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference"
  ☆101Updated last month
• tonyzhang617 / nomad-dist
  ☆38Updated last year
• vllm-project / flash-attention
  Fast and memory-efficient exact attention
  ☆111Updated last week
• ByteDance-Seed / StragglerAnalysis
  ☆51Updated 9 months ago
• ruipeterpan / marconi
  Artifact for "Marconi: Prefix Caching for the Era of Hybrid LLMs" [MLSys '25 Outstanding Paper Award, Honorable Mention]
  ☆49Updated 11 months ago
• WukLab / preble
  Stateful LLM Serving
  ☆95Updated 10 months ago
• abcdabcd987 / libfabric-efa-demo
  ☆77Updated last year
• meta-pytorch / KernelAgent
  Autonomous GPU Kernel Generation via Deep Agents
  ☆228Updated this week
• EfficientMoE / MoE-Infinity
  PyTorch library for cost-effective, fast and easy serving of MoE models.
  ☆279Updated this week
• efeslab / fiddler
  [ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration
  ☆260Updated last year
• infinigence / Semi-PD
  A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.
  ☆123Updated last month
• Azure / msccl
  Microsoft Collective Communication Library
  ☆66Updated last year
• infinigence / FUSCO
  High-performance distributed data shuffling (all-to-all) library for MoE training and inference
  ☆111Updated last month
• DataStates / datastates-llm
  LLM checkpointing for DeepSpeed/Megatron
  ☆24Updated 2 months ago
• UChi-JCL / CacheGen
  ☆150Updated last year
• eunomia-bpf / cupti-tutorial
  Tutorials for NVIDIA CUPTI samples
  ☆50Updated 3 months ago
• Azure / msccl-executor-nccl
  ☆47Updated last year
• flashinfer-ai / flashinfer-bench
  Building the Virtuous Cycle for AI-driven LLM Systems
  ☆151Updated this week
• ROCm / TransformerEngine
  ☆60Updated this week
• abhibambhaniya / GenZ-LLM-Analyzer
  LLM Inference analyzer for different hardware platforms
  ☆100Updated 2 months ago
• Infrawaves / DeepEP_ibrc_dual-ports_multiQP
  Aims to implement dual-port and multi-qp solutions in deepEP ibrc transport
  ☆73Updated 8 months ago
• Hsword / SpotServe
  SpotServe: Serving Generative Large Language Models on Preemptible Instances
  ☆135Updated last year
• LLMServe / SwiftTransformer
  High performance Transformer implementation in C++.
  ☆150Updated last year