zenrran4nlp / Awesome-LLM-Inference-ServingLinks
☆47Updated 8 months ago
Alternatives and similar repositories for Awesome-LLM-Inference-Serving
Users that are interested in Awesome-LLM-Inference-Serving are comparing it to the libraries listed below
Sorting:
- LLM Serving Performance Evaluation Harness☆82Updated 10 months ago
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆135Updated last year
- KV cache compression for high-throughput LLM inference☆148Updated 10 months ago
- Cascade Speculative Drafting☆32Updated last year
- [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts☆40Updated last year
- ☆48Updated last year
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆175Updated last year
- [NeurIPS 2025] Simple extension on vLLM to help you speed up reasoning model without training.☆215Updated 7 months ago
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)☆112Updated 9 months ago
- Easy, Fast, and Scalable Multimodal AI☆82Updated last week
- ☆63Updated 7 months ago
- ☆102Updated last year
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆42Updated last year
- Compression for Foundation Models☆34Updated 5 months ago
- ☆38Updated last year
- Code repo for "CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs".☆16Updated last year
- The evaluation framework for training-free sparse attention in LLMs☆108Updated 2 months ago
- The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.☆52Updated last year
- The official repo for "LLoCo: Learning Long Contexts Offline"☆118Updated last year
- Self-host LLMs with LMDeploy and BentoML☆21Updated this week
- Beyond KV Caching: Shared Attention for Efficient LLMs☆20Updated last year
- Vocabulary Parallelism☆24Updated 9 months ago
- Official Implementation of APB (ACL 2025 main Oral)☆32Updated 10 months ago
- [CoLM'25] The official implementation of the paper <MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression>☆153Updated last month
- A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM☆178Updated last week
- (ACL 2025 oral) SCOPE: Optimizing KV Cache Compression in Long-context Generation☆33Updated 7 months ago
- Repo hosting codes and materials related to speeding LLMs' inference using token merging.☆37Updated 2 months ago
- This repository is the official implementation of "Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE"☆36Updated 2 months ago
- Official Implementation of SAM-Decoding: Speculative Decoding via Suffix Automaton☆39Updated 10 months ago
- [NeurIPS 2025] Scaling Speculative Decoding with Lookahead Reasoning☆56Updated 2 months ago