High-performance single-GPU inference for selected model checkpoints and GPUs.
☆3,206Oct 29, 2026Updated this week
Alternatives and similar repositories for ninfer
Users that are interested in ninfer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on loca…☆35,851Updated this week
- Hyper optimised Qwen3.8-27B inference on one RTX 3090: ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux bu…☆484Oct 16, 2026Updated last week
- KVMem virtualizes KV memory across inference backends to make ultra-long-context local AI accessible on limited hardware.☆1,187Updated this week
- Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when …☆2,413Oct 21, 2026Updated last week
- High-performance single-GPU inference for selected model checkpoints and GPUs.☆184Sep 30, 2026Updated last month
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Qwen3.8-27B on one RTX 4090: full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention pre…☆222Updated this week
- C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer☆161Updated this week
- Strata fork: Qwen3.8-Flash-Next 125B MoE in NVFP4 on one RTX 20-50 card (12 GB+) and 64 GB of RAM or more. Setup installs our GPTQ quants…☆201Updated this week
- Native Windows port of NInfer engine for RTX 5090. Features Qwen3.8-27B with QUASAR and NInfer models, MTP/DFlash2 with vision and 262,14…☆78Updated this week
- Project Maya - run GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE) and MiniMax-M3.1-Flash (428B MoE) privately on your own …☆2,190Updated this week
- FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.☆14,817Updated this week
- One line of NInfer for the Blackwell, Ada Lovelace, Ampere, Turing and Volta NVIDIA GPUs, consolidated from the forks that carry it: GGUF…☆167Updated this week
- NInfer port for Qwen 3.8 27B on RTX 4090☆183Sep 9, 2026Updated last month
- ☆492Oct 20, 2026Updated last week
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- High-performance single-GPU inference for selected model checkpoints and GPUs.☆24Oct 13, 2026Updated 2 weeks ago
- LLM speculative inference server for heterogeneous hardware & consumer GPUs☆2,961Updated this week
- llama.cpp fork with additional SOTA quants and improved performance☆3,350Updated this week
- High-performance single-GPU inference for selected model checkpoints and GPUs.☆255Sep 14, 2026Updated last month
- An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs☆1,763Updated this week
- Local inference engine for Qwen3.8-Flash-Next on NVIDIA Blackwell☆300Updated this week
- Community recipes for serving LLMs on RTX 3090/4090/5090 CUDA gpus. Multi-engine (vLLM, sglang, llama.cpp and other custom engines) and m…☆2,454Updated this week
- LLM Inference Engine for Metal, CUDA and Vulkan.☆1,532Updated this week
- V100 / SM70-focused vLLM engineering fork for modern LLM inference.☆1,421Updated this week
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- Durable local inference for Oh My Pi: NInfer + Qwen3.8 27B on one RTX 3090/4090/5090, with restart-resumable OpenAI Responses state.☆47Updated this week
- ninfer-ext: Forked for speed.☆27Oct 22, 2026Updated last week
- KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM☆1,177Updated this week
- A reproducible local LLM benchmarking platform with versioned datasets, deterministic scoring, and auditable reports.☆177Updated this week
- High-performance single-GPU inference for RTX 5090 Mobile.☆20Oct 4, 2026Updated 3 weeks ago
- Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthrop…☆1,136Updated this week
- Unlock P2P comms between consumer NVIDIA GPUs☆680Updated this week
- GVS5H: Five Qwen3.8-27B Models Match Claude Fable 5 on LiveCodeBench Hard | Fable 5 Level Coding for a Fifth the Price - or on a Single…☆1,110Updated this week
- A local inference engine for Apple silicon, built around the model.☆1,929Updated this week
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Qwen3.8-27B-NVFP4 on a Single RTX 5090 (vLLM)☆100Aug 16, 2026Updated 2 months ago
- Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc☆6,108Updated this week
- An all-in-one, pure C++ inference engine for audio models, powered by ggml. Supports TTS, STT, VAD, voice conversion, music generation, a…☆3,848Updated this week
- Hand-written NVFP4 W4A16 CUDA kernels for Volta☆176Aug 19, 2026Updated 2 months ago
- Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦☆44,896Updated this week
- Your GeForce runs AI at half speed. This fixes it for 20/30/40 series cards. Up to 2× FP16 / BF16 / TF32 tensor throughput☆12,884Updated this week
- LLM inference in C/C++☆983Updated this week