Alternatives and similar repositories for vllm-router:

Users that are interested in vllm-router are comparing it to the libraries listed below

• zhaochenyang20 / ModelServer
  Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang
  ☆47Updated 5 months ago
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆70Updated 2 months ago
• IsaacRe / vllm-kvcompress
  KV cache compression for high-throughput LLM inference
  ☆126Updated 2 months ago
• vllm-project / dashboard
  vLLM performance dashboard
  ☆27Updated 11 months ago
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆131Updated 10 months ago
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆112Updated 4 months ago
• GreenBitAI / green-bit-llm
  A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.
  ☆82Updated last month
• InternLM / turbomind
  ☆78Updated 3 weeks ago
• nyunAI / PruneGPT
  ☆53Updated 10 months ago
• bentoml / llm-bench
  ☆49Updated 4 months ago
• LMCache / lmcache-vllm
  The driver for LMCache core to run in vLLM
  ☆36Updated 2 months ago
• bentoml / BentoLMDeploy
  Self-host LLMs with LMDeploy and BentoML
  ☆18Updated last month
• neuralmagic / AutoFP8
  ☆185Updated 6 months ago
• LLM360 / crystalcoder-data-prep
  Data preparation code for CrystalCoder 7B LLM
  ☆44Updated 11 months ago
• triton-inference-server / vllm_backend
  ☆241Updated this week
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆65Updated last year
• feifeibear / LLMRoofline
  Compare different hardware platforms via the Roofline Model for LLM inference tasks.
  ☆93Updated last year
• intel / llm-on-ray
  Pretrain, finetune and serve LLMs on Intel platforms with Ray
  ☆125Updated 2 weeks ago
• wejoncy / QLLM
  A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ/VPTQ, and export to onnx/onnx-runtime easily.
  ☆165Updated 2 weeks ago
• FreedomIntelligence / FastLLM
  Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];
  ☆36Updated last year
• EmbeddedLLM / vllm
  vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆87Updated this week
• foundation-model-stack / bamba
  Train, tune, and infer Bamba model
  ☆88Updated 3 months ago
• tyler-griggs / melange-release
  ☆45Updated 9 months ago
• run-ai / runai-model-streamer
  ☆190Updated last week
• run-ai / llmperf
  ☆54Updated 6 months ago
• NVIDIA / workbench-llamafactory
  This is an NVIDIA AI Workbench example project that demonstrates an end-to-end model development workflow using Llamafactory.
  ☆54Updated 6 months ago
• wdlctc / headinfer
  ☆43Updated 2 weeks ago
• hetailang / SqueezeAttention
  ☆37Updated 6 months ago
• MooreThreads / TurboRAG
  ☆74Updated 4 months ago
• samchaineau / llm_slerp_generation
  Repo hosting codes and materials related to speeding LLMs' inference using token merging.
  ☆36Updated 11 months ago