romsto / Speculative-DecodingLinks
Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.
☆66Updated 6 months ago
Alternatives and similar repositories for Speculative-Decoding
Users that are interested in Speculative-Decoding are comparing it to the libraries listed below
Sorting:
- Explorations into some recent techniques surrounding speculative decoding☆269Updated 6 months ago
- Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)☆282Updated 2 months ago
- [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆303Updated 5 months ago
- ☆256Updated last year
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆311Updated last month
- Code associated with the paper **Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding**☆190Updated 4 months ago
- 📰 Must-read papers and blogs on Speculative Decoding ⚡️☆800Updated last week
- 📰 Must-read papers on KV Cache Compression (constantly updating 🤗).☆459Updated this week
- [ICLR 2025] Palu: Compressing KV-Cache with Low-Rank Projection☆122Updated 4 months ago
- [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"☆85Updated last year
- Official PyTorch implementation of DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024)☆221Updated 3 months ago
- ☆52Updated 6 months ago
- Implementation of Speculative Sampling as described in "Accelerating Large Language Model Decoding with Speculative Sampling" by Deepmind☆98Updated last year
- LLM KV cache compression made easy☆520Updated this week
- Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.☆321Updated 3 months ago
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆310Updated 11 months ago
- [NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization☆359Updated 10 months ago
- Awesome LLM pruning papers all-in-one repository with integrating all useful resources and insights.☆91Updated 6 months ago
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆297Updated 7 months ago
- ☆328Updated last year
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆163Updated 11 months ago
- [ICML 2024 Oral] Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs☆108Updated 2 months ago
- Official Pytorch Implementation of "Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity"☆69Updated 11 months ago
- A family of compressed models obtained via pruning and knowledge distillation☆343Updated 7 months ago
- [ICLR2025 Spotlight] MagicPIG: LSH Sampling for Efficient LLM Generation☆219Updated 6 months ago
- The official implementation of the paper "Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques (TMLR)".☆71Updated 3 months ago
- Awesome list for LLM pruning.☆232Updated 6 months ago
- Fast inference from large lauguage models via speculative decoding☆762Updated 10 months ago
- Official Implementation of "Learning Harmonized Representations for Speculative Sampling" (HASS)☆39Updated 3 months ago
- [ICLR 2025] PEARL: Parallel Speculative Decoding with Adaptive Draft Length☆89Updated 2 months ago