vllm-project / dashboard
vLLM performance dashboard
☆18Updated 6 months ago
Related projects ⓘ
Alternatives and complementary repositories for dashboard
- Boosting 4-bit inference kernels with 2:4 Sparsity☆51Updated 2 months ago
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆79Updated this week
- KV cache compression for high-throughput LLM inference☆87Updated this week
- Materials for learning SGLang☆105Updated this week
- A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ, and export to onnx/onnx-runtime easily.☆150Updated last month
- ☆35Updated 2 weeks ago
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆38Updated 10 months ago
- PyTorch bindings for CUTLASS grouped GEMM.☆53Updated 3 weeks ago
- Decoding Attention is specially optimized for multi head attention (MHA) using CUDA core for the decoding stage of LLM inference.☆25Updated 2 weeks ago
- Odysseus: Playground of LLM Sequence Parallelism☆57Updated 5 months ago
- [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts☆34Updated 8 months ago
- ☆158Updated last month
- ☆96Updated last month
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆147Updated 4 months ago
- Quantized Attention on GPU☆30Updated 2 weeks ago
- An easy-to-use package for implementing SmoothQuant for LLMs☆83Updated 6 months ago
- Production ready LLM model compression/quantization toolkit with accelerated inference support for both cpu/gpu via HF, vLLM, and SGLang.☆125Updated this week
- ☆55Updated 5 months ago
- LLM Serving Performance Evaluation Harness☆56Updated 2 months ago
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆24Updated last week
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)☆76Updated last month
- Code for Palu: Compressing KV-Cache with Low-Rank Projection☆57Updated this week
- ☆47Updated 2 months ago
- ☆42Updated 6 months ago
- Official PyTorch implementation of FlatQuant: Flatness Matters for LLM Quantization☆63Updated last week
- GPTQ inference TVM kernel☆36Updated 6 months ago
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆126Updated 5 months ago
- Manages vllm-nccl dependency☆17Updated 5 months ago
- QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference☆112Updated 8 months ago
- PyTorch bindings for CUTLASS grouped GEMM.☆68Updated 4 months ago