smart-lty / ParallelSpeculativeDecoding
The official code for paper "parallel speculative decoding with adaptive draft length."
☆24Updated 2 months ago
Related projects ⓘ
Alternatives and complementary repositories for ParallelSpeculativeDecoding
- Multi-Candidate Speculative Decoding☆28Updated 6 months ago
- Implementation of Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting☆44Updated 4 months ago
- ☆64Updated 3 months ago
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆147Updated 4 months ago
- The Official Implementation of Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference☆38Updated this week
- ☆42Updated 7 months ago
- QAQ: Quality Adaptive Quantization for LLM KV Cache☆42Updated 7 months ago
- Official PyTorch implementation of FlatQuant: Flatness Matters for LLM Quantization☆63Updated last week
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)☆76Updated last month
- ☆31Updated this week
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆202Updated 2 weeks ago
- [ACL 2024] A novel QAT with Self-Distillation framework to enhance ultra low-bit LLMs.☆84Updated 6 months ago
- Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)☆188Updated 3 weeks ago
- Code associated with the paper **Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding**☆138Updated 5 months ago
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆78Updated this week
- ☆188Updated 6 months ago
- ATC23 AE☆43Updated last year
- MagicPIG: LSH Sampling for Efficient LLM Generation☆58Updated 3 weeks ago
- Triton implementation of Flash Attention2.0☆22Updated last year
- ☆42Updated 6 months ago
- Code for Palu: Compressing KV-Cache with Low-Rank Projection☆57Updated this week
- Boosting 4-bit inference kernels with 2:4 Sparsity☆51Updated 2 months ago
- OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure☆18Updated 3 months ago
- 📰 Must-read papers on KV Cache Compression (constantly updating 🤗).☆136Updated this week
- Awesome list for LLM quantization☆127Updated this week
- Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models☆36Updated 2 weeks ago
- PyTorch bindings for CUTLASS grouped GEMM.☆53Updated 3 weeks ago
- Official Repo for SparseLLM: Global Pruning of LLMs (NeurIPS 2024)☆37Updated this week
- PyTorch bindings for CUTLASS grouped GEMM.☆68Updated 4 months ago
- ☆35Updated 3 months ago