huggingface / inference-benchmarkerLinks
Inference server benchmarking tool
☆67Updated last month
Alternatives and similar repositories for inference-benchmarker
Users that are interested in inference-benchmarker are comparing it to the libraries listed below
Sorting:
- ☆99Updated this week
- Benchmark suite for LLMs from Fireworks.ai☆75Updated 3 weeks ago
- ArcticTraining is a framework designed to simplify and accelerate the post-training process for large language models (LLMs)☆105Updated this week
- Load compute kernels from the Hub☆139Updated this week
- vLLM performance dashboard☆30Updated last year
- Manage scalable open LLM inference endpoints in Slurm clusters☆258Updated 10 months ago
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆116Updated 6 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆263Updated 7 months ago
- ☆121Updated last month
- A safetensors extension to efficiently store sparse quantized tensors on disk☆117Updated this week
- IBM development fork of https://github.com/huggingface/text-generation-inference☆60Updated 3 weeks ago
- vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs☆86Updated this week
- KV cache compression for high-throughput LLM inference☆129Updated 4 months ago
- ☆215Updated this week
- Google TPU optimizations for transformers models☆112Updated 4 months ago
- ☆46Updated last week
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆153Updated 7 months ago
- Cray-LM unified training and inference stack.☆22Updated 4 months ago
- Repo hosting codes and materials related to speeding LLMs' inference using token merging.☆36Updated last year
- 👷 Build compute kernels☆44Updated this week
- 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.☆137Updated 10 months ago
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆301Updated last week
- A collection of all available inference solutions for the LLMs☆88Updated 3 months ago
- The Batched API provides a flexible and efficient way to process multiple requests in a batch, with a primary focus on dynamic batching o…☆136Updated last week
- Comparison of Language Model Inference Engines☆217Updated 5 months ago
- Docker image NVIDIA GH200 machines - optimized for vllm serving and hf trainer finetuning☆42Updated 3 months ago
- Data preparation code for Amber 7B LLM☆91Updated last year
- ☆53Updated 8 months ago
- vLLM adapter for a TGIS-compatible gRPC server.☆30Updated this week
- Easy and Efficient Quantization for Transformers☆198Updated 3 months ago