Alternatives and similar repositories for demo

Users that are interested in demo are comparing it to the libraries listed below

• LMCache / lmcache-vllm
  The driver for LMCache core to run in vLLM
  ☆51Updated 7 months ago
• yale-sys / prompt-cache
  Modular and structured prompt caching for low-latency LLM inference
  ☆100Updated 10 months ago
• MooreThreads / TurboRAG
  ☆81Updated 10 months ago
• project-etalon / etalon
  LLM Serving Performance Evaluation Harness
  ☆79Updated 7 months ago
• tyler-griggs / melange-release
  ☆47Updated last year
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆127Updated 9 months ago
• ovg-project / kvcached
  kvcached: Elastic KV cache for dynamic GPU sharing and efficient multi-LLM inference.
  ☆93Updated this week
• hao-ai-lab / MuxServe
  ☆71Updated last year
• microsoft / ParrotServe
  [OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable
  ☆183Updated last year
• flexflow / flexflow-serve
  FlexFlow Serve: Low-Latency, High-Performance LLM Serving
  ☆58Updated last week
• kvcache-ai / TrEnv-X
  ☆54Updated last week
• WukLab / preble
  Stateful LLM Serving
  ☆85Updated 6 months ago
• YaoJiayi / CacheBlend
  ☆137Updated 2 months ago
• microsoft / RetrievalAttention
  Scalable long-context LLM decoding that leverages sparsity—by treating the KV cache as a vector storage system.
  ☆81Updated last week
• argonne-lcf / LLM-Inference-Bench
  LLM-Inference-Bench
  ☆52Updated 2 months ago
• eddiegaoo / Apt-Serve
  ☆17Updated 3 months ago
• InternLM / turbomind
  ☆95Updated 6 months ago
• microsoft / tokenweave
  Efficient Compute-Communication Overlap for Distributed LLM Inference
  ☆43Updated 2 weeks ago
• timlee0212 / SiDA-MoE
  Code for MLSys 2024 Paper "SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models"
  ☆21Updated last year
• IsaacRe / vllm-kvcompress
  KV cache compression for high-throughput LLM inference
  ☆138Updated 7 months ago
• zcli-charlie / Awesome-KV-Cache
  ☆77Updated 11 months ago
• vllm-project / dashboard
  vLLM performance dashboard
  ☆35Updated last year
• vllm-project / flash-attention
  Fast and memory-efficient exact attention
  ☆94Updated last week
• UChi-JCL / CacheGen
  ☆130Updated 11 months ago
• ScalingIntelligence / hydragen
  Hydragen: High-Throughput LLM Inference with Shared Prefixes
  ☆41Updated last year
• Hsword / SpotServe
  SpotServe: Serving Generative Large Language Models on Preemptible Instances
  ☆129Updated last year
• zenrran4nlp / Awesome-LLM-Inference-Serving
  ☆42Updated 4 months ago
• PKU-SEC-Lab / HybriMoE
  [DAC'25] Official implement of "HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference"
  ☆71Updated 3 months ago
• stepfun-ai / StepMesh
  ☆295Updated this week
• hao-ai-lab / vllm-ltr
  [NeurIPS 2024] Efficient LLM Scheduling by Learning to Rank
  ☆59Updated 10 months ago