Hyper optimised Qwen3.8-27B inference on one RTX 3090: ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds.
☆485Oct 16, 2026Updated 2 weeks ago
Alternatives and similar repositories for ninfer-3090
Users that are interested in ninfer-3090 are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when …☆2,413Oct 21, 2026Updated last week
- Run Qwen3.8-27B at 140 tok/s on a single RTX 3090, up to 2x llama.cpp. Fast local LLM inference server with OpenAI- and Anthropic-compati…☆65Updated this week
- Community recipes for serving LLMs on RTX 3090/4090/5090 CUDA gpus. Multi-engine (vLLM, sglang, llama.cpp and other custom engines) and m…☆2,460Updated this week
- High-performance single-GPU inference for selected model checkpoints and GPUs.☆3,215Updated this week
- NInfer port for Qwen 3.8 27B on RTX 4090☆183Sep 9, 2026Updated last month
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- llama.cpp fork for significantly improved performance on Ampere (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token ge…☆202Updated this week
- One line of NInfer for the Blackwell, Ada Lovelace, Ampere, Turing and Volta NVIDIA GPUs, consolidated from the forks that carry it: GGUF…☆171Updated this week
- Qwen3.8-27B on a single RTX 4090 (sm_89): Cinference/NInfer port with MTP speculative decoding and K8V4 KV cache☆14Oct 20, 2026Updated last week
- Qwen3.8-27B on one RTX 4090: full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention pre…☆222Updated this week
- Benchmarks, compose files, and findings for running Qwen3.6 (27B dense + 35B-A3B MoE) on 4× RTX 3090. vLLM/AWQ vs llama.cpp/GGUF, MTP spe…☆21Aug 15, 2026Updated 2 months ago
- Unlock P2P comms between consumer NVIDIA GPUs☆680Updated this week
- A cmpunlocker fork with working p2p on SYS-4028GR-TRT2☆26Aug 8, 2026Updated 2 months ago
- On-device voice isolation, wake words, voice activity and voiceprints. In the browser (WebAssembly) and in Python, with plugins for LiveK…☆34Updated this week
- Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on loca…☆36,386Updated this week
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- NInfer SM80 port for NVIDIA CMP 170HX☆29Aug 22, 2026Updated 2 months ago
- CMP 170HX technical wiki: silicon, firmware, the community unlock, procedures and open problems. 55 pages.☆75Aug 22, 2026Updated 2 months ago
- A tool to unlobotomize your NVIDIA card!☆817Updated this week
- KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM☆1,177Updated this week
- LLM speculative inference server for heterogeneous hardware & consumer GPUs☆2,964Updated this week
- C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer☆164Updated this week
- Qwen3.8-27B EXL3 (4.00 bpw) + DFlash2 speculative decoding for ExLlamaV3, validated at 262k context on a 24 GB RTX 3090☆25Sep 20, 2026Updated last month
- SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35…☆133Sep 30, 2026Updated last month
- A tool to unlobotomize your NVIDIA card!☆31Sep 6, 2026Updated last month
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- Project Maya - run GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE) and MiniMax-M3.1-Flash (428B MoE) privately on your own …☆2,190Updated this week
- GLM-5.3-Flash on 4× NVIDIA CMP 170HX with vLLM: up to 482.6 tok/s single-user, up to 943.7 tok/s at 8 users, 262K context, TP4 or PP4, on…☆87Updated this week
- An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs☆1,763Updated this week
- Durable local inference for Oh My Pi: NInfer + Qwen3.8 27B on one RTX 3090/4090/5090, with restart-resumable OpenAI Responses state.☆47Updated this week
- Qwen3.8 Flash Next AWQ INT4 4x3090 v1: 262K, Vision, FP8 QSA KV and CUDA Graphs on 4x RTX 3090☆16Aug 30, 2026Updated 2 months ago
- 🚿 Llaminar: an LLM inferencing engine in C++.☆47Updated this week
- High-performance single-GPU inference for selected model checkpoints and GPUs.☆24Oct 13, 2026Updated 2 weeks ago
- Hand-written NVFP4 W4A16 CUDA kernels for Volta☆178Aug 19, 2026Updated 2 months ago
- Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,86…☆130Oct 15, 2026Updated 2 weeks ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- DeepSeek-V4-Flash-0731 on 4x CMP 170HX (sm_80): 98 tok/s decode, ~5300 tok/s prefill. DSpark speculative decoding under pipeline parallel…☆118Aug 13, 2026Updated 2 months ago
- LLM quantization optimizer☆23Aug 20, 2026Updated 2 months ago
- Tuning and qualification harness for the NVIDIA CMP 170HX (GA100). Measures, gates and recovers clock/voltage settings, and treats a comp…☆96Sep 28, 2026Updated last month
- Qwen3.8-27B INT8 + DFlash2 on a rented CMP 170HX — failure analysis (private GHCR image prevented benchmark)☆17Aug 30, 2026Updated 2 months ago
- Xolotl (formerly Strata-DSV4.1): run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 90 GB RAM) at up…☆419Updated this week
- Strata minus the bullshit. clone, rsync to prod, compile and go☆196Oct 24, 2026Updated last week
- vLLM 0.32 fork for consumer GPUs: 3 x 262K sessions on 4x RTX 3090, Qwen 3.8 Flash-Next at 184 tok/s, up to 17 on bigger cards. Seven che…☆31Oct 22, 2026Updated last week