Hyper optimised Qwen3.8-27B inference on one RTX 3090: ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds.
☆523Oct 31, 2026Updated 2 weeks ago
Alternatives and similar repositories for ninfer-3090
Users that are interested in ninfer-3090 are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when …☆2,751Nov 10, 2026Updated last week
- Run Qwen3.8-27B at 140 tok/s on a single RTX 3090, up to 2x llama.cpp. Fast local LLM inference server with OpenAI- and Anthropic-compati…☆82Updated this week
- Community recipes for serving LLMs on RTX 3090/4090/5090 CUDA gpus. Multi-engine (vLLM, sglang, llama.cpp and other custom engines) and m…☆2,507Updated this week
- High-performance single-GPU inference for selected model checkpoints and GPUs.☆3,387Updated this week
- llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ to…☆402Updated this week
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Qwen3.8-27B on a single RTX 4090 (sm_89): Cinference/NInfer port with MTP speculative decoding and K8V4 KV cache☆15Nov 1, 2026Updated 2 weeks ago
- One line of NInfer for the Blackwell, Ada Lovelace, Ampere, Turing and Volta NVIDIA GPUs, consolidated from the forks that carry it: GGUF…☆234Updated this week
- Qwen3.8-27B on one RTX 4090: full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention pre…☆239Oct 31, 2026Updated 2 weeks ago
- NInfer port for Qwen 3.8 27B on RTX 4090☆187Updated this week
- Qwen3.8-27B on a single 24 GB GPU at 1.8–2× llama.cpp's decode speed. Bit-exact MTP speculative decoding, exact conversation caching, and…☆23Nov 11, 2026Updated last week
- A cmpunlocker fork with working p2p on SYS-4028GR-TRT2☆27Aug 8, 2026Updated 3 months ago
- Unlock P2P comms between consumer NVIDIA GPUs☆735Oct 26, 2026Updated 3 weeks ago
- Run GLM-5.3 (744B), GLM-5.3-Flash (321B), DeepSeek-V4.1-Flash (748B) and MiniMax-M3.1-Flash (428B) on your own GPUs - one engine, one das…☆3,061Updated this week
- KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM☆1,199Updated this week
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- CMP 170HX technical wiki: silicon, firmware, the community unlock, procedures and open problems. 55 pages.☆80Aug 22, 2026Updated 2 months ago
- An attempt to use Nvidia CMP cards for Machine Learning.☆33Jun 14, 2026Updated 5 months ago
- Lucebox inference engine for heterogeneous hardware & consumer GPUs☆3,018Updated this week
- SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35…☆133Sep 30, 2026Updated last month
- A tool to unlobotomize your NVIDIA card!☆925Oct 28, 2026Updated 3 weeks ago
- Benchmarks, compose files, and findings for running Qwen3.6 (27B dense + 35B-A3B MoE) on 4× RTX 3090. vLLM/AWQ vs llama.cpp/GGUF, MTP spe…☆21Aug 15, 2026Updated 3 months ago
- Known-good vLLM recipe: Qwen3.8-27B-Int8 + MTP speculative decoding on NVIDIA CMP 170HX 64GB (Hynix) — 84 tok/s, 262k ctx☆13Oct 8, 2026Updated last month
- Fast Qwen3.8-27B for one RTX 3090. 200k context, 100+ tok/s. Custom CUDA decode engine, OpenAI-compatible API.☆42Updated this week
- vLLM 0.33 fork for consumer GPUs: 3 x 262K sessions on Ampere,Ada and more! Qwen 3.8 Flash-Next at 207 tok/s, up to 17 on bigger cards. S…☆51Nov 12, 2026Updated last week
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Your GeForce runs AI at half speed. This fixes it for 20/30/40 series cards. Up to 2× FP16 / BF16 / TF32 tensor throughput☆20,461Updated this week
- An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs☆1,840Updated this week
- ☆119Apr 28, 2026Updated 6 months ago
- High-performance single-GPU inference for selected model checkpoints and GPUs.☆25Oct 13, 2026Updated last month
- Lightnovel generator using GUI☆13Sep 15, 2026Updated 2 months ago
- Qwen3.8-27B EXL3 (4.00 bpw) + DFlash2 speculative decoding for ExLlamaV3, validated at 262k context on a 24 GB RTX 3090☆25Sep 20, 2026Updated 2 months ago
- NInfer SM80 port for NVIDIA CMP 170HX☆29Aug 22, 2026Updated 2 months ago
- Experimental llama.cpp fork for inference research and development☆910Nov 13, 2026Updated last week
- llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived f…☆35Nov 13, 2026Updated last week
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer☆308Updated this week
- One llama.cpp flag unlocks +33-39% decode speed for Qwen3.8-27B on consumer GPUs. The MTP head already ships inside your GGUF. Recipe, pa…☆382Sep 26, 2026Updated last month
- Reliable model swapping for any local AI server. Chosen by thousands of users. Batteries included. If we don't have it, you probably don…☆6,272Updated this week
- Web front end for making video and music on your own ComfyUI — MiniMax H3, Music 3, SeedVR2 upscaling and FLUX Kontext edits. Deploys to …☆20Updated this week
- A tool to unlobotomize your NVIDIA card!☆31Sep 6, 2026Updated 2 months ago
- Tuning and qualification harness for the NVIDIA CMP 170HX (GA100). Measures, gates and recovers clock/voltage settings, and treats a comp…☆99Sep 28, 2026Updated last month
- Durable local inference for Oh My Pi: NInfer + Qwen3.8 27B on one RTX 3090/4090/5090, with restart-resumable OpenAI Responses state.☆51Updated this week
Are these results useful?