NVIDIA / kvpressLinks
LLM KV cache compression made easy
☆876Updated last week
Alternatives and similar repositories for kvpress
Users that are interested in kvpress are comparing it to the libraries listed below
Sorting:
- [MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Se…☆810Updated 11 months ago
- Efficient LLM Inference over Long Sequences☆394Updated 7 months ago
- Accelerating MoE with IO and Tile-aware Optimizations☆569Updated 3 weeks ago
- KernelBench: Can LLMs Write GPU Kernels? - Benchmark + Toolkit with Torch -> CUDA (+ more DSLs)☆792Updated 2 weeks ago
- 📰 Must-read papers on KV Cache Compression (constantly updating 🤗).☆649Updated 4 months ago
- Perplexity GPU Kernels☆554Updated 3 months ago
- [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads☆524Updated 11 months ago
- ArcticInference: vLLM plugin for high-throughput, low-latency inference☆384Updated last week
- [NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization☆404Updated last year
- A throughput-oriented high-performance serving framework for LLMs☆945Updated 3 months ago
- Miles is an enterprise-facing reinforcement learning framework for LLM and VLM post-training, forked from and co-evolving with slime.☆830Updated this week
- Applied AI experiments and examples for PyTorch☆315Updated 5 months ago
- kernels, of the mega variety☆665Updated last week
- Fast low-bit matmul kernels in Triton☆427Updated last week
- [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆356Updated 2 months ago
- A Quirky Assortment of CuTe Kernels☆781Updated this week
- Train speculative decoding models effortlessly and port them smoothly to SGLang serving.☆676Updated this week
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆279Updated 2 months ago
- ☆957Updated 3 months ago
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆370Updated 6 months ago
- Dynamic Memory Management for Serving LLMs without PagedAttention☆457Updated 8 months ago
- A safetensors extension to efficiently store sparse quantized tensors on disk☆238Updated this week
- Cataloging released Triton kernels.☆292Updated 5 months ago
- Load compute kernels from the Hub☆397Updated this week
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆475Updated this week
- FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.☆1,005Updated last year
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆356Updated 2 weeks ago
- Scalable toolkit for efficient model reinforcement☆1,293Updated this week
- Code for Neurips24 paper: QuaRot, an end-to-end 4-bit inference of large language models.☆480Updated last year
- Block Diffusion for Ultra-Fast Speculative Decoding☆459Updated this week