A high-throughput and memory-efficient inference and serving engine for LLMs (Windows build & kernels)
☆662Oct 11, 2026Updated this week
Alternatives and similar repositories for vllm-windows
Users that are interested in vllm-windows are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- FlashInfer: Kernel Library for LLM Serving (Windows build & kernels)☆19Jul 1, 2026Updated 3 months ago
- Native Windows vLLM: 0.27.1 Latest; 0.29.0 Python 3.14/CUDA 13.2 release (CPU TorchAudio), plus Python 3.13/CUDA 13.0 prerelease. PyTorch…☆59Updated this week
- Patched native-Windows build of vLLM. Three Windows-specific fixes (CPU-relay for Gloo, Qwen3 reasoning parser, wildcard model name) on t…☆28May 8, 2026Updated 5 months ago
- Fork of the Triton language and compiler for Windows support and easy installation☆1,950Feb 18, 2026Updated 7 months ago
- One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry.☆231May 14, 2026Updated 4 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- llama.cpp fork with additional SOTA quants and improved performance☆3,293Updated this week
- An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs☆1,641Updated this week
- Triton with Windows support☆275Updated this week
- Pynini for Windows☆26Apr 11, 2025Updated last year
- Quantized Attention that achieves speedups of 2.1-3.1x and 2.7-5.1x compared to FlashAttention2 and xformers, respectively, without lossi…☆158Sep 24, 2026Updated 2 weeks ago
- Fast fp16-fp8 mixed precision matmul on RDNA3/3.5 GPUs without native fp8☆39Oct 3, 2026Updated last week
- KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM☆1,151Updated this week
- This is a pre-built wheel of Triton 3.3.0 for Windows with Nvidia only + Proton☆44May 18, 2025Updated last year
- Deepspeed windows information☆44Mar 9, 2024Updated 2 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Pre-compiled Python whl for Flash-attention, SageAttention, NATTEN, xFormer etc☆876Sep 7, 2026Updated last month
- KoboldCpp - Run GGUF AI models locally with a KoboldAI UI. One File. Zero Install.☆11,988Updated this week
- Official ConvRot implementation. A plug-and-play, convolution-like rotation module enabling efficient W4A4 quantization for diffusion mod…☆32Jul 3, 2026Updated 3 months ago
- Fast and memory-efficient exact attention☆944Dec 9, 2025Updated 10 months ago
- Flash Attention WHL Builds for Windows☆16May 30, 2025Updated last year
- A graphical tool to simplify building and installing DeepSpeed 0.15.x or later on Windows systems.☆21Apr 26, 2025Updated last year
- ik_llama.cpp's Thireus fork with release builds for macOS/Windows/Ubuntu CPU, Vulkan and CUDA☆180Updated this week
- [ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models☆3,959Sep 6, 2026Updated last month
- Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc☆5,924Updated this week
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- A high-throughput and memory-efficient inference and serving engine for LLMs☆93,564Updated this week
- Thireus's fork of llama.cpp with Cuda 12.8 and 13.3 release builds and Windows patch for loading more .gguf shards + llama-sweep-bench☆30Updated this week
- Advanced CLI diffusion inference/training suite based on Musubi Tuner☆39Apr 15, 2026Updated 5 months ago
- Lower Precision Floating Point Operations☆86Feb 22, 2026Updated 7 months ago
- A transformers implementation of csm-streaming☆30May 16, 2025Updated last year
- LLM inference in C/C++☆2,426Updated this week
- Llama Server Launcher (llama.cpp/ik_llama) GUI☆129Sep 28, 2026Updated last week
- Omni inference in C/C++☆276Updated this week
- [ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-t…☆81Jan 19, 2026Updated 8 months ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- LLM inference in C/C++☆130,827Updated this week
- Produce your own Dynamic 3.0 Quants and achieve optimum accuracy & SOTA quantization performance! Input a target size and the toolchain w…☆170Sep 11, 2026Updated last month
- LLM speculative inference server for heterogeneous hardware & consumer GPUs☆2,911Updated this week
- gguf (GPT-Generated Unified Format) connector☆62Sep 24, 2026Updated 2 weeks ago
- Experimental llama.cpp fork for inference research and development☆887Updated this week
- Fork of SageAttention for Windows wheels and easy installation☆1,057Updated this week
- ☆190Aug 30, 2026Updated last month