Alternatives and similar repositories for benchmark:

Users that are interested in benchmark are comparing it to the libraries listed below

• bentoml / llm-bench
  ☆49Updated 4 months ago
• run-ai / llmperf
  ☆54Updated 6 months ago
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆112Updated 4 months ago
• LMCache / lmcache-vllm
  The driver for LMCache core to run in vLLM
  ☆36Updated 2 months ago
• IsaacRe / vllm-kvcompress
  KV cache compression for high-throughput LLM inference
  ☆125Updated 2 months ago
• project-etalon / etalon
  LLM Serving Performance Evaluation Harness
  ☆75Updated last month
• neuralmagic / AutoFP8
  ☆185Updated 6 months ago
• anyscale / llm-continuous-batching-benchmarks
  ☆117Updated last year
• snowflakedb / ArcticTraining
  ArcticTraining is a framework designed to simplify and accelerate the post-training process for large language models (LLMs)
  ☆64Updated this week
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 10 months ago
• dust-tt / llama-ssp
  Experiments on speculative sampling with Llama models
  ☆125Updated last year
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆97Updated 3 weeks ago
• vllm-project / dashboard
  vLLM performance dashboard
  ☆26Updated 11 months ago
• triton-inference-server / vllm_backend
  ☆241Updated this week
• jaymody / speculative-sampling
  Simple implementation of Speculative Sampling in NumPy for GPT-2.
  ☆93Updated last year
• GreenBitAI / green-bit-llm
  A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.
  ☆82Updated last month
• IBM / text-generation-inference
  IBM development fork of https://github.com/huggingface/text-generation-inference
  ☆60Updated 3 months ago
• ppl-ai / pplx-kernels
  Perplexity GPU Kernels
  ☆185Updated last week
• opendatahub-io / vllm-tgis-adapter
  vLLM adapter for a TGIS-compatible gRPC server.
  ☆25Updated this week
• microsoft / sarathi-serve
  A low-latency & high-throughput serving engine for LLMs
  ☆337Updated 2 months ago
• AniZpZ / smoothquant
  [ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  ☆11Updated last year
• EmbeddedLLM / vllm
  vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆87Updated this week
• IST-DASLab / Sparse-Marlin
  Boosting 4-bit inference kernels with 2:4 Sparsity
  ☆72Updated 7 months ago
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆65Updated last year
• huggingface / inference-benchmarker
  Inference server benchmarking tool
  ☆48Updated last week
• apple / ml-recurrent-drafter
  ☆205Updated 2 months ago
• AdrianBZG / LLM-distributed-finetune
  Tune efficiently any LLM model from HuggingFace using distributed training (multiple GPU) and DeepSpeed. Uses Ray AIR to orchestrate the …
  ☆56Updated last year
• mlc-ai / llm-perf-bench
  ☆117Updated 11 months ago
• fw-ai / llama-cuda-graph-example
  Example of applying CUDA graphs to LLaMA-v2
  ☆12Updated last year
• neuralmagic / nm-vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆262Updated 6 months ago