Alternatives and similar repositories for llm-inference

Users that are interested in llm-inference are comparing it to the libraries listed below

• bentoml / llm-bench
  ☆54Updated 7 months ago
• zhaochenyang20 / ModelServer
  Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang
  ☆53Updated 7 months ago
• InternLM / turbomind
  ☆86Updated 2 months ago
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆133Updated last year
• neuralmagic / AutoFP8
  ☆194Updated last month
• feifeibear / Odysseus-Transformer
  Odysseus: Playground of LLM Sequence Parallelism
  ☆70Updated last year
• chu-tianxiang / QuIP-for-all
  QuIP quantization
  ☆54Updated last year
• intel / llm-on-ray
  Pretrain, finetune and serve LLMs on Intel platforms with Ray
  ☆129Updated last month
• anyscale / llm-continuous-batching-benchmarks
  ☆119Updated last year
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆76Updated 2 weeks ago
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆128Updated last year
• NetEase-FuXi / EETQ
  Easy and Efficient Quantization for Transformers
  ☆199Updated 4 months ago
• vllm-project / flash-attention
  Fast and memory-efficient exact attention
  ☆74Updated last week
• Dao-AILab / grouped-latent-attention
  ☆114Updated 3 weeks ago
• usyd-fsalab / fp6_llm
  An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).
  ☆252Updated 7 months ago
• jaymody / speculative-sampling
  Simple implementation of Speculative Sampling in NumPy for GPT-2.
  ☆95Updated last year
• efeslab / Atom
  [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  ☆310Updated 11 months ago
• MDK8888 / vllmini
  A minimal implementation of vllm.
  ☆43Updated 10 months ago
• IST-DASLab / SparseFinetuning
  Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry
  ☆42Updated last year
• deepspeedai / DeepSpeed-Kernels
  ☆72Updated 2 months ago
• IST-DASLab / Sparse-Marlin
  Boosting 4-bit inference kernels with 2:4 Sparsity
  ☆79Updated 9 months ago
• shreyansh26 / Speculative-Sampling
  Implementation of Speculative Sampling as described in "Accelerating Large Language Model Decoding with Speculative Sampling" by Deepmind
  ☆98Updated last year
• madsys-dev / deepseekv2-profile
  ☆141Updated 3 months ago
• zms1999 / SmartMoE
  A MoE impl for PyTorch, [ATC'23] SmartMoE
  ☆63Updated last year
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆116Updated 6 months ago
• Ascend / AscendSpeed
  ☆79Updated last year
• ModelTC / awesome-lm-system
  Summary of system papers/frameworks/codes/tools on training or serving large model
  ☆57Updated last year
• IsaacRe / vllm-kvcompress
  KV cache compression for high-throughput LLM inference
  ☆131Updated 4 months ago
• feifeibear / LLMRoofline
  Compare different hardware platforms via the Roofline Model for LLM inference tasks.
  ☆100Updated last year
• gpu-mode / ring-attention
  ring-attention experiments
  ☆144Updated 8 months ago