TurboQuant+ KV cache compression for vLLM. 3.8x smaller KV cache, same conversation quality. Fused CUDA kernels with automatic PyTorch fallback.
☆82Sep 19, 2026Updated last month
Alternatives and similar repositories for turboquant-vllm
Users that are interested in turboquant-vllm are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Benchmark your local LLMs with real evaluation suites. Not vibes - actual scores.☆31Updated this week
- Tested MiaAI-based DeepSeek V4 Flash speed profile for two ASUS GX10s: pinned patch, settings, and benchmark helper.☆15Sep 5, 2026Updated last month
- llama.cpp fork with TQ3_1S/4S CUDA kernels — 3.5-bit WHT quantization achieving Q4s quality at 10% smaller size. Based on RaBitQ-inspired…☆233Oct 13, 2026Updated last week
- General Matrix Multiplication using NVIDIA Tensor Cores☆29Jan 25, 2025Updated last year
- ☆28Mar 31, 2026Updated 6 months ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Universal AI computer-control KVM (REST API + MCP) that drives any machine as plain USB hardware — nothing installed on the target. Built…☆20Oct 6, 2026Updated 2 weeks ago
- Nano vLLM☆26Aug 11, 2025Updated last year
- This project is specifically developed for V100, based on lmdeploy 0.12.1, and supports mainstream open-source models from Q4 2025 to Q1 …☆22Mar 18, 2026Updated 7 months ago
- ☆13Aug 31, 2026Updated last month
- NEW ROCmfp4 format for llama.cpp☆158Jun 13, 2026Updated 4 months ago
- Native app wrapper for Chatto, adding system tray, notifications, auto-start, deep link support☆14Oct 7, 2026Updated 2 weeks ago
- VENDORIZED in lucebox-hub. Fork of llama.cpp, ggml graph for lucebox inference engine☆30Jul 8, 2026Updated 3 months ago
- SGLang for cool dudes using V100☆80Oct 5, 2026Updated 3 weeks ago
- Deploy concurrent Hermes Agent workers on unified-memory GPUs (GB10, DGX Spark) for maximum total tok/s. Profile-isolated, kanban-coordin…☆83Jul 18, 2026Updated 3 months ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- vLLM TurboQuant☆618Jun 25, 2026Updated 4 months ago
- Run SGLang on 1-to-N NVIDIA DGX Spark servers (single Spark, 2 via direct cable, or 3+ via switched fabric) to serve or benchmark LLMs☆22May 1, 2026Updated 5 months ago
- Experiments for the spectral neuron paper☆23Sep 5, 2026Updated last month
- Open quantization tooling for TurboQuant-style low-bit LLM releases, stock GGUF deployment, and Apple Silicon runtime experiments.☆10May 5, 2026Updated 5 months ago
- Browse, search, and recover files from OpenCode AI coding sessions from TUI or web. View diffs, explore snapshots, and download any proje…☆16Jan 4, 2026Updated 9 months ago
- A Rust compiler backend and runtime that compiles Rust kernels to many NPU/GPU accelerators via MLIR — vendor-agnostic (Ascend, Trainium,…☆20Updated this week
- HTTP Proxy that allows you to define the IP address at connect time☆20Jan 11, 2026Updated 9 months ago
- ☆22Oct 6, 2026Updated 2 weeks ago
- Generates kid-friendly weather forecasts suitable for display on a large monitor. Uses OpenWeatherMap and a local or hosted LLM.☆17Feb 28, 2026Updated 7 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Learn about PTX instructions ldmatrix and stmatrix☆12Sep 5, 2025Updated last year
- 一个Turing专用vLLM分支，通过整合FlashInfer 0.6.18（PR #3526）与SM75后端门槛补丁（PR #47949），持续同步vLLM主线最新版本☆35Oct 11, 2026Updated 2 weeks ago
- Electron speech-to-speech app for your voice calls based on 100% locally run AI models☆36Jul 23, 2025Updated last year
- 🐕 Kelpie shepherds long-running jobs through to completion on interruptible hardware, coordinating with the Kelpie API☆11Sep 17, 2026Updated last month
- This is a repository for documenting the setup and performance of MI100s in popular inference engines.☆19Oct 8, 2026Updated 2 weeks ago
- Portable personal advisor OS built from Agent Skills☆12Aug 12, 2026Updated 2 months ago
- pytorch+bert实现的意图识别与槽位填充☆11May 30, 2023Updated 3 years ago
- a QA bot on contents of given docs 用所给文档进行问答的聊天机器人☆12Apr 20, 2023Updated 3 years ago
- A FastAPI and MCP server connected to Supabase table for managing, testing, and programmatically accessing LLM System Instructions, Syste…☆14Updated this week
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- Probe how GPT-n performs on statutory reasoning☆10Sep 17, 2024Updated 2 years ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆38Updated this week
- ☆17Sep 6, 2026Updated last month
- Low-latency IPC library for building persistent agentic tool servers (LLM inference, TTS, vector search, browser automation) over named p…☆15Sep 15, 2026Updated last month
- CLI, MCP server, and TypeScript library that extracts text, layout, and page images from PDFs for AI agents — when extraction goes wrong,…☆24Updated this week
- ☆68Feb 24, 2026Updated 8 months ago
- One-click Chrome extension that harvests Reddit tabs, categorizes content with AI, and builds a searchable knowledge base. Uses Anthropic…☆12Mar 1, 2026Updated 7 months ago