vllm-project / guidellmLinks
Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
☆596Updated this week
Alternatives and similar repositories for guidellm
Users that are interested in guidellm are comparing it to the libraries listed below
Sorting:
- Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM☆2,019Updated this week
- ArcticInference: vLLM plugin for high-throughput, low-latency inference☆267Updated this week
- ☆298Updated last week
- LLMPerf is a library for validating and benchmarking LLMs☆1,019Updated 9 months ago
- A throughput-oriented high-performance serving framework for LLMs☆893Updated 2 weeks ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆266Updated 11 months ago
- ☆255Updated this week
- Comparison of Language Model Inference Engines☆229Updated 9 months ago
- A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM☆50Updated this week
- Inference server benchmarking tool☆112Updated 5 months ago
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆315Updated last week
- Advanced Quantization Algorithm for LLMs and VLMs, with support for CPU, Intel GPU, CUDA and HPU.☆647Updated this week
- A collection of all available inference solutions for the LLMs☆91Updated 7 months ago
- llm-d enables high-performance distributed LLM inference on Kubernetes☆1,806Updated this week
- OpenAI compatible API for TensorRT LLM triton backend☆215Updated last year
- ☆111Updated 3 weeks ago
- Checkpoint-engine is a simple middleware to update model weights in LLM inference engines☆751Updated this week
- Serverless LLM Serving for Everyone.☆552Updated 3 weeks ago
- OME is a Kubernetes operator for enterprise-grade management and serving of Large Language Models (LLMs)☆279Updated last week
- ☆59Updated last year
- Common recipes to run vLLM☆146Updated this week
- An Open Source Toolkit For LLM Distillation☆729Updated 2 months ago
- An innovative library for efficient LLM inference via low-bit quantization☆350Updated last year
- ArcticTraining is a framework designed to simplify and accelerate the post-training process for large language models (LLMs)☆219Updated this week
- Materials for learning SGLang☆590Updated last month
- LLM KV cache compression made easy☆623Updated this week
- LLM model quantization (compression) toolkit with hw acceleration support for Nvidia CUDA, AMD ROCm, Intel XPU and Intel/AMD/Apple CPU vi…☆806Updated this week
- VPTQ, A Flexible and Extreme low-bit quantization algorithm☆659Updated 5 months ago
- vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization☆1,815Updated this week
- [NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention…☆1,133Updated this week