Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
☆479Nov 7, 2026Updated this week
Alternatives and similar repositories for Xolotl
Users that are interested in Xolotl are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Run GLM-5.3-Flash (321B), DeepSeek-V4.1-Flash (748B) and MiniMax-M3.1-Flash (428B) on your own GPUs - one engine, one dashboard, OpenAI- …☆2,632Updated this week
- Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith …☆906Updated this week
- Local inference engine for Qwen3.8-Flash-Next on NVIDIA Blackwell☆744Updated this week
- DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) on 2x RTX 3090 + 128 GB RAM. Rust + CUDA, T16 quantization. 40+ tok/s measured with exp…☆93Updated this week
- Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on loca…☆41,381Updated this week
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Almond: a multi-model MoE engine for AMD RX 7900 XTX (ROCm/HIP) - DeepSeek-V4.1-Flash, GLM-5.3-Flash, Qwen3.8-Flash-Next☆53Updated this week
- Delta Inference Engine: DeepSeek-V4.1-Flash on one NVIDIA GPU, natively on Windows (OpenAI-compatible server)☆75Updated this week
- C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer☆280Updated this week
- Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when …☆2,601Updated this week
- Stable, high-throughput inference server for multi-GPU NVIDIA V100 (SM70): Qwen3.8 Flash-Next with AWQ / NVFP4, multimodal input, CPU/dis…☆48Updated this week
- Quasar is an inference engine for Qwen3.6-27B-MTP / Qwen3.8-27B on consumer NVIDIA cards as well as Apple Silicon, in the spirit of antir…☆52Updated this week
- Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and h…☆289Updated this week
- Unlock P2P comms between consumer NVIDIA GPUs☆697Oct 26, 2026Updated last week
- High-performance single-GPU inference for selected model checkpoints and GPUs.☆3,283Updated this week
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- A fork of https://github.com/Niko1221/Strata optimized for my 3090+5070 Ti setup, see docs/COMPARISON.md. Renamed to Lamina.☆59Updated this week
- Your GeForce runs AI at half speed. This fixes it for 20/30/40 series cards. Up to 2× FP16 / BF16 / TF32 tensor throughput☆17,787Oct 29, 2026Updated last week
- Strata for a PC with two NVIDIA GPUs and 32 GB of RAM: the low-RAM mode on a layer split, both cards working at the same time. 143 tok/s …☆84Oct 6, 2026Updated last month
- Qwen3.8-Flash-Next 180B NVFP4 @ 231 tok/s on one RTX PRO 6000 Blackwell (sm120) - sglang patches, launchers, full optimization analysis☆30Aug 31, 2026Updated 2 months ago
- RocketLLama. Fork off LLama cpp that newly scales 2-20x now also on big models not fully fitting in vram on even just 2 gpus☆16Updated this week
- 两张 V100 跑 Qwen3.8-27B 到 100 tok/s（186K 上下文）：重写 sm70 注意力内核 + 多卡张量并行。Fork of geoffwatts/ninfer-v100.☆18Sep 28, 2026Updated last month
- KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM☆1,187Updated this week
- Hand-written NVFP4 W4A16 CUDA kernels for Volta☆181Aug 19, 2026Updated 2 months ago
- An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs☆1,800Updated this week
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- LLM inference in C/C++☆473Updated this week
- CPU-GPU hybrid inference for DeepSeek-V4 on NVIDIA SM80+ (A100/RTX 4090 etc.), forked from yhfgyyf/vllm-deepseek-v4-sm89.☆61Sep 16, 2026Updated last month
- llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ to…☆267Updated this week
- A frontier model edits another model's weights by hand (Gemma 4 12B, EN->IT tense). Plus an awareness experiment: tampering Qwen3.8-27B i…☆28Updated this week
- NVIDIA Linux open GPU with P2P support☆43Oct 26, 2026Updated last week
- ☆119Apr 28, 2026Updated 6 months ago
- Pure-Rust MoE inference engine for Qwen3.5/3.6/3.8 GGUF. CPU+GPU hybrid experts, NVRTC CUDA kernels, OpenAI-compatible server. 2x llama.c…☆131Updated this week
- Strata minus the bullshit. clone, rsync to prod, compile and go☆221Updated this week
- DeepSeek-V4.1-Flash EXL3 on one 24 GB RTX 3090 + DDR4 + NVMe: staged-DMA prefill, AVX2 CPU expert tier, elastic VRAM expert cache, Engram…☆155Oct 12, 2026Updated 3 weeks ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Fast fork of llama.cpp designed for Qwen3.8-Flash-Next☆350Updated this week
- The fastest inference engine for Qwen3.8-27B on the NVIDIA RTX 5090: up to 500 tokens/s for one agent and up to 2,000 tokens/s for many, …☆61Updated this week
- Custom HIP inference engine for AMD MI50/MI60 (gfx906)☆19Oct 4, 2026Updated last month
- More MoEs: a fork of Strata (Qwen3.8-Flash-Next, Qwen3.6/Ornith and more on one GPU + system RAM)☆25Updated this week
- A tool to unlobotomize your NVIDIA card!☆31Sep 6, 2026Updated 2 months ago
- Source-available C++/CUDA inference engine for Qwen3.8-27B INT4 on 3x RTX 3090☆43Updated this week
- Builds vLLM for ROCm gfx906 using Nix☆18Aug 29, 2026Updated 2 months ago