From-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) for LLM KV cache compression. 5x compression at 3-bit with 99.5% attention fidelity.
☆1,027Apr 23, 2026Updated 2 months ago
Alternatives and similar repositories for turboquant-pytorch
Users that are interested in turboquant-pytorch are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration☆1,662Mar 27, 2026Updated 3 months ago
- Rust implementation of TurboQuant, PolarQuant, and QJL — zero-overhead vector quantization for semantic search and KV cache compression (…☆29May 31, 2026Updated last month
- ☆6,992Jun 26, 2026Updated 2 weeks ago
- vLLM TurboQuant☆610Jun 25, 2026Updated 2 weeks ago
- DFlash: Block Diffusion for Flash Speculative Decoding☆5,468May 10, 2026Updated 2 months ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- LLM inference in C/C++☆2,097Updated this week
- Running a big model on a small laptop☆3,980Mar 19, 2026Updated 3 months ago
- Fast LLM speculative inference server for consumer hardware.☆2,656Updated this week
- KV cache compression via block-diagonal rotation. Beats TurboQuant: better PPL (6.91 vs 7.07), 28% faster decode, 5.3x faster prefill, 44…☆1,027Apr 23, 2026Updated 2 months ago
- First open-source KVTC implementation (NVIDIA, ICLR 2026) -- 8-32x KV cache compression via PCA + adaptive quantization + entropy coding☆21Apr 17, 2026Updated 2 months ago
- A vector index built on TurboQuant, written in Rust with Python bindings☆12,690Jun 10, 2026Updated last month
- SpectralQuant: Calibrated Eigenbasis Rotation and Water-Filled Bit Allocation for KV-Cache Compression☆197May 15, 2026Updated last month
- 🤗 ml-intern: an open-source ML engineer that reads papers, trains models, and ships ML models☆10,648Updated this week
- A theoretical reconstruction of the Claude Mythos architecture, built from first principles using the available research literature.☆14,681May 23, 2026Updated last month
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- Autoresearch for GPU kernels. Give it any PyTorch model, go to sleep, wake up to optimized Triton kernels.☆1,458Mar 19, 2026Updated 3 months ago
- Library for Google's Turboquant Algorithm☆67Mar 29, 2026Updated 3 months ago
- Kakao Mobility MCP Server for directions and transit information☆11Sep 14, 2025Updated 10 months ago
- eLLM can infer LLM on CPUs faster than on GPUs☆427Updated this week
- DeepSpec: a full-stack codebase for training and evaluating speculative decoding algorithms☆6,638Updated this week
- Training framework for Large Behavioral Models☆28Sep 17, 2025Updated 9 months ago
- Meta-Harness: 76.4% on Terminal-Bench 2.0 (Claude Opus 4.6)☆1,142Mar 26, 2026Updated 3 months ago
- TurboQuant+ KV cache compression for vLLM. 3.8x smaller KV cache, same conversation quality. Fused CUDA kernels with automatic PyTorch fa…☆74Updated this week
- DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm☆18,521Jul 3, 2026Updated last week
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Fast and memory-efficient exact attention☆24,452Updated this week
- Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦☆11,798Updated this week
- Self-referential self-improving agents that can optimize for any computable task☆2,637May 9, 2026Updated 2 months ago
- TokenSpeed is a speed-of-light LLM inference engine.☆1,589Updated this week
- [ICML 2025] CommVQ: Commutative Vector Quantization for KV Cache Compression☆27Sep 2, 2025Updated 10 months ago
- An agent-managed museum exhibit, built in Rust with Gajae-Code / LazyCodex — developed and maintained with no human intervention.☆194,754Jun 26, 2026Updated 2 weeks ago
- The agent that grows with you☆214,747Updated this week
- ☆119Feb 25, 2026Updated 4 months ago
- Rust implementation of Google's TurboQuant algorithm for vector quantization☆36Mar 25, 2026Updated 3 months ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- 26m function call model that runs on incredibly small devices☆3,042Jul 1, 2026Updated last week
- A curated collection of AI video prompting resources, featuring official guides, prompt templates, cinematic techniques, and audio-vi…☆65Jan 4, 2026Updated 6 months ago
- A kernel library written in tilelang☆1,642Apr 23, 2026Updated 2 months ago
- The best-benchmarked open-source AI memory system. And it's free.☆57,326Updated this week
- "OpenSpace: The Quality-First Skill Hub for AI Agents" -- https://open-space.cloud/☆6,732Updated this week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆86,251Updated this week
- A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative…☆3,230Updated this week