Alternatives and similar repositories for llm-on-ray:

Users that are interested in llm-on-ray are comparing it to the libraries listed below

• run-ai / llmperf
  ☆52Updated 4 months ago
• bentoml / llm-bench
  ☆45Updated 2 months ago
• triton-inference-server / vllm_backend
  ☆223Updated this week
• LMCache / lmcache-vllm
  The driver for LMCache core to run in vLLM
  ☆26Updated last week
• AlibabaPAI / llumnix
  Efficient and easy multi-instance LLM serving
  ☆288Updated this week
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆65Updated this week
• tyler-griggs / melange-release
  ☆43Updated 7 months ago
• project-etalon / etalon
  LLM Serving Performance Evaluation Harness
  ☆66Updated 5 months ago
• microsoft / sarathi-serve
  A low-latency & high-throughput serving engine for LLMs
  ☆308Updated 2 weeks ago
• anyscale / llm-continuous-batching-benchmarks
  ☆117Updated 10 months ago
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆61Updated 10 months ago
• sgl-project / sgl-learning-materials
  Materials for learning SGLang
  ☆253Updated last week
• feifeibear / LLMRoofline
  Compare different hardware platforms via the Roofline Model for LLM inference tasks.
  ☆93Updated 11 months ago
• neuralmagic / AutoFP8
  ☆172Updated 4 months ago
• HabanaAI / vllm-fork
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆56Updated this week
• microsoft / vattention
  Dynamic Memory Management for Serving LLMs without PagedAttention
  ☆285Updated this week
• run-ai / runai-model-streamer
  ☆157Updated last week
• EmbeddedLLM / vllm
  vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆88Updated this week
• AdrianBZG / LLM-distributed-finetune
  Tune efficiently any LLM model from HuggingFace using distributed training (multiple GPU) and DeepSpeed. Uses Ray AIR to orchestrate the …
  ☆55Updated last year
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆107Updated 2 months ago
• cli99 / llm-analysis
  Latency and Memory Analysis of Transformer Models for Training and Inference
  ☆388Updated 3 months ago
• microsoft / vidur
  A large-scale simulation framework for LLM inference
  ☆321Updated 2 months ago
• mlc-ai / llm-perf-bench
  ☆117Updated 9 months ago
• yale-sys / prompt-cache
  Modular and structured prompt caching for low-latency LLM inference
  ☆87Updated 3 months ago
• InternLM / turbomind
  ☆65Updated 2 months ago
• efeslab / Atom
  [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  ☆293Updated 7 months ago
• OpenPPL / ppl.llm.serving
  ☆127Updated last month
• vectorch-ai / ScaleLLM
  A high-performance inference system for large language models, designed for production environments.
  ☆411Updated this week
• deepspeedai / DeepSpeed-Kernels
  ☆59Updated last week
• LMCache / LMCache
  Making Long-Context LLM Inference 10x Faster and 10x Cheaper
  ☆459Updated this week