shadowpa0327 / Palu
Code for Palu: Compressing KV-Cache with Low-Rank Projection
☆57Updated last week
Related projects ⓘ
Alternatives and complementary repositories for Palu
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆149Updated 4 months ago
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆203Updated 3 weeks ago
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆79Updated this week
- 16-fold memory access reduction with nearly no loss☆58Updated 2 weeks ago
- Boosting 4-bit inference kernels with 2:4 Sparsity☆51Updated 2 months ago
- ☆96Updated 2 months ago
- ☆189Updated 6 months ago
- [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆243Updated last month
- MagicPIG: LSH Sampling for Efficient LLM Generation☆59Updated 3 weeks ago
- QAQ: Quality Adaptive Quantization for LLM KV Cache☆42Updated 7 months ago
- Implementation of Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting☆44Updated 4 months ago
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆278Updated 4 months ago
- ☆42Updated 7 months ago
- PyTorch bindings for CUTLASS grouped GEMM.☆53Updated 3 weeks ago
- Odysseus: Playground of LLM Sequence Parallelism☆57Updated 5 months ago
- This repo contains the source code for: Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs☆32Updated 3 months ago
- [ICML 2024 Oral] Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs☆83Updated 3 months ago
- Fast Matrix Multiplications for Lookup Table-Quantized LLMs☆187Updated this week
- Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMs☆74Updated 5 months ago
- Materials for learning SGLang☆110Updated this week
- PyTorch library for cost-effective, fast and easy serving of MoE models.☆103Updated 3 months ago
- The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.☆39Updated last month
- A sparse attention kernel supporting mix sparse patterns☆58Updated last month
- 📰 Must-read papers on KV Cache Compression (constantly updating 🤗).☆143Updated this week
- Official PyTorch implementation of FlatQuant: Flatness Matters for LLM Quantization☆66Updated last week
- [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts☆34Updated 8 months ago
- Activation-aware Singular Value Decomposition for Compressing Large Language Models☆49Updated last month
- KV cache compression for high-throughput LLM inference☆87Updated last week
- ☆47Updated 2 months ago
- Simple and fast low-bit matmul kernels in CUDA / Triton☆147Updated this week