bytedance / ShadowKV
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
☆126Updated 3 weeks ago
Related projects ⓘ
Alternatives and complementary repositories for ShadowKV
- [COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding☆230Updated 2 months ago
- Official Code for "SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression"☆95Updated last month
- MIXQ: Taming Dynamic Outliers in Mixed-Precision Quantization by Online Prediction☆83Updated 3 weeks ago
- An acceleration library that supports arbitrary bit-width combinatorial quantization operations☆226Updated last month
- The Official Implementation of Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference☆39Updated this week
- 16-fold memory access reduction with nearly no loss☆58Updated last week
- ☆42Updated 7 months ago
- MagicPIG: LSH Sampling for Efficient LLM Generation☆59Updated 3 weeks ago
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆203Updated 3 weeks ago
- A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of …☆106Updated 4 months ago
- A sparse attention kernel supporting mix sparse patterns☆58Updated last month
- Official PyTorch implementation of FlatQuant: Flatness Matters for LLM Quantization☆66Updated last week
- Implementation of Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting☆44Updated 4 months ago
- ☆64Updated 4 months ago
- QAQ: Quality Adaptive Quantization for LLM KV Cache☆42Updated 7 months ago
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆278Updated 4 months ago
- ☆43Updated 6 months ago
- Quantized Attention on GPU☆31Updated this week
- Support mixed-precsion inference with vllm☆97Updated 2 weeks ago
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆79Updated this week
- ☆51Updated last month
- SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models☆13Updated last month
- Code for Palu: Compressing KV-Cache with Low-Rank Projection☆57Updated last week
- The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.☆39Updated last month
- 📰 Must-read papers on KV Cache Compression (constantly updating 🤗).☆143Updated this week
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆149Updated 4 months ago
- ☆38Updated last month
- Materials for learning SGLang☆110Updated this week
- Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.☆114Updated 2 weeks ago
- [ACL 2024] A novel QAT with Self-Distillation framework to enhance ultra low-bit LLMs.☆85Updated 6 months ago