NVIDIA / kvpressLinks
LLM KV cache compression made easy
☆799Updated this week
Alternatives and similar repositories for kvpress
Users that are interested in kvpress are comparing it to the libraries listed below
Sorting:
- Accelerating MoE with IO and Tile-aware Optimizations☆542Updated last week
- [MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Se…☆802Updated 10 months ago
- Efficient LLM Inference over Long Sequences☆393Updated 6 months ago
- kernels, of the mega variety☆648Updated 3 months ago
- A throughput-oriented high-performance serving framework for LLMs☆937Updated 2 months ago
- Perplexity GPU Kernels☆553Updated 2 months ago
- Fast low-bit matmul kernels in Triton☆423Updated last month
- 📰 Must-read papers on KV Cache Compression (constantly updating 🤗).☆640Updated 3 months ago
- KernelBench: Can LLMs Write GPU Kernels? - Benchmark + Toolkit with Torch -> CUDA (+ more DSLs)☆763Updated this week
- [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads☆519Updated 11 months ago
- ArcticInference: vLLM plugin for high-throughput, low-latency inference☆375Updated this week
- A Quirky Assortment of CuTe Kernels☆749Updated this week
- Miles is an enterprise-facing reinforcement learning framework for large-scale MoE post-training and production workloads, forked from an…☆714Updated this week
- Applied AI experiments and examples for PyTorch☆314Updated 4 months ago
- [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆348Updated 2 months ago
- Train speculative decoding models effortlessly and port them smoothly to SGLang serving.☆626Updated this week
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆365Updated 6 months ago
- [NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization☆400Updated last year
- Dynamic Memory Management for Serving LLMs without PagedAttention☆457Updated 7 months ago
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆278Updated last month
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆352Updated 8 months ago
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆469Updated last week
- ☆949Updated 2 months ago
- A safetensors extension to efficiently store sparse quantized tensors on disk☆234Updated this week
- FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.☆987Updated last year
- Cataloging released Triton kernels.☆287Updated 4 months ago
- Code for Neurips24 paper: QuaRot, an end-to-end 4-bit inference of large language models.☆476Updated last year
- ☆578Updated 3 months ago
- Load compute kernels from the Hub☆376Updated this week
- ☆233Updated last year