mit-han-lab / deepcompressor
Model Compression Toolbox for Large Language Models and Diffusion Models
☆161Updated this week
Related projects ⓘ
Alternatives and complementary repositories for deepcompressor
- Code for Neurips24 paper: QuaRot, an end-to-end 4-bit inference of large language models.☆281Updated 3 months ago
- An algorithm for static activation quantization of LLMs☆67Updated this week
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆195Updated last week
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆277Updated 4 months ago
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆240Updated last month
- QQQ is an innovative and hardware-optimized W4A8 quantization solution for LLMs.☆76Updated last month
- Fast Matrix Multiplications for Lookup Table-Quantized LLMs☆184Updated last month
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆146Updated 4 months ago
- QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference☆112Updated 8 months ago
- ☆183Updated 6 months ago
- Simple and fast low-bit matmul kernels in CUDA / Triton☆137Updated this week
- This repository contains integer operators on GPUs for PyTorch.☆181Updated last year
- An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).☆196Updated 2 weeks ago
- QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving☆434Updated this week
- A sparse attention kernel supporting mix sparse patterns☆53Updated 3 weeks ago
- ☆46Updated last month
- ☆95Updated last month
- An easy-to-use package for implementing SmoothQuant for LLMs☆82Updated 5 months ago
- Official PyTorch implementation of FlatQuant: Flatness Matters for LLM Quantization☆58Updated this week
- Code repo for the paper "SpinQuant LLM quantization with learned rotations"☆151Updated this week
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆70Updated last week
- Boosting 4-bit inference kernels with 2:4 Sparsity☆51Updated 2 months ago
- [NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization☆303Updated 2 months ago
- Applied AI experiments and examples for PyTorch☆160Updated last week
- ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation☆31Updated 2 months ago
- Official implementation of the ICLR 2024 paper AffineQuant☆21Updated 7 months ago
- Fast Hadamard transform in CUDA, with a PyTorch interface☆108Updated 5 months ago
- DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads☆348Updated last week
- Code for Palu: Compressing KV-Cache with Low-Rank Projection☆54Updated this week
- A parallelism VAE avoids OOM for high resolution image generation☆40Updated last month