LLM inference decode throughput benchmark with Rich TUI dashboard. Measures token generation speed across concurrency levels and context lengths. Supports SGLang and vLLM engines.
☆120Oct 13, 2026Updated last week
Alternatives and similar repositories for llm-inference-bench
Users that are interested in llm-inference-bench are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Docker images for LLM inference (SGLang + vLLM) on NVIDIA Blackwell GPUs (SM120, CUDA 13.2)☆92Updated this week
- Archived: b12x development has moved to FlashInfer. See README for migration and PR-porting guidance.☆268Oct 6, 2026Updated 2 weeks ago
- RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink☆1,148Updated this week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆42Updated this week
- Sharing benchmarks from DGX stations☆27Sep 11, 2026Updated last month
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ☆37Apr 26, 2026Updated 5 months ago
- Linux anonymity / OPSEC toolkit — Tor-by-default with a kill switch, leak verification, and pluggable upstreams. Authorized use only.☆23Sep 2, 2026Updated last month
- KiChad — a Codex-integrated downstream of KiCad☆22Jul 24, 2026Updated 2 months ago
- Profile repo — categorized index of NVFP4 model releases, DGX Spark inference stacks, Apple Silicon MLX builds, the voice-AI stack, and t…☆41Oct 6, 2026Updated 2 weeks ago
- Open-source macOS AI computer use and native GUI automation on Apple silicon. Jev + OmniParser CoreML + Apple Vision OCR. Bring your own …☆24Sep 18, 2026Updated last month
- LLMs take the path of least resistance. Pre-Reasoning hands any model a map of the problem first, so it unlocks the knowledge it already …☆19Sep 21, 2026Updated last month
- Headless / multi-user serving-mode for the DGX Spark: free unified memory for vLLM by paring back the desktop. Companion to qwen3.5-122B-…☆19Jun 29, 2026Updated 3 months ago
- 1.0 Beta · oMLX 0.6.3rc2 Dual-ANE+CPU · Qwen3.8-27B abliterated oQ4e · Mac Studio M3 Ultra☆45Aug 23, 2026Updated 2 months ago
- DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.☆56Updated this week
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- Custom inference recipes for NVIDIA DGX Spark — Qwen 122B DFlash hybrid☆89Updated this week
- GLM-5.3-Flash (NVFP4) at TP=4 across 4x ASUS GX10 / GB10 - vLLM + mentat instead of Ray, 524k context, DFlash2. Minimal working recipe.☆90Updated this week
- Deploy concurrent Hermes Agent workers on unified-memory GPUs (GB10, DGX Spark) for maximum total tok/s. Profile-isolated, kanban-coordin…☆83Jul 18, 2026Updated 3 months ago
- ☆25Sep 3, 2026Updated last month
- Serve Qwen3.8-Flash-Next (turboderp ExLlamaV3 pack) on one NVIDIA DGX Spark with exllamav3☆61Oct 14, 2026Updated last week
- This project is an adapted version of Seth-McKilla's project with the newest Wagmi version.☆17Apr 21, 2023Updated 3 years ago
- Minimal read-only OS image for GB10 (DGX Spark / ASUS GX10) boxes☆90Updated this week
- Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quan…☆140Oct 15, 2026Updated last week
- GLM-5.3-Flash (NVFP4) on 2x NVIDIA DGX Spark - vLLM TP2, 262K context, MTP. World-first deploy recipe: 7 day-0 bugs found and fixed, patc…☆237Sep 30, 2026Updated 3 weeks ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- This is an experimental fork of Salvatore Sanfilippo's `antirez/ds4` project: ```text https://github.com/antirez/ds4 ``` This checkout …☆28Sep 23, 2026Updated last month
- Optimized cross-platform expansion of autoresearch.☆18Mar 23, 2026Updated 7 months ago
- Autonomous self-improving 4x DGX Spark (GB10) MoA stack + LoRA loop (DSV4F router, Qwen3.6/Omni/TwoTower/Gemma). Hermes MoA routing, ~90%…☆24Aug 17, 2026Updated 2 months ago
- AEON vLLM Ultimate — vLLM 0.27.1 built from source for DGX Spark / Blackwell (sm_121a/GB10). DSpark quantized Markov heads, DFlash SWA on…☆146Oct 6, 2026Updated 2 weeks ago
- Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, a…☆394Updated this week
- antirez/ds4-style hybrid quant DeepSeek V4 Flash on a single DGX Spark via vLLM☆17May 11, 2026Updated 5 months ago
- GLM-5.3-Flash (EXL3 4bpw + DFlash2) on 2x NVIDIA DGX Spark: one-shot installer, 33-74 tok/s c1, 1.3M+ context, vision☆43Sep 2, 2026Updated last month
- Fast, deterministic verifier and gate for AI agents: exact math, claim and citation checks, shell/write policy, hash-chained step ledger …☆44Updated this week
- Pi extension that tracks bash tool token usage with live stats, grouping, and export☆23Feb 10, 2026Updated 8 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Serve the official Ling-3.0-flash INT4 on one NVIDIA DGX Spark, 38.7 tok/s, faster than the community GGUF. Recipe, watchdog, benchmarks.☆26Aug 29, 2026Updated last month
- A local commit gate: runs the linters your repo configures, and sends changed code to an LLM for review☆13Updated this week
- Qwen3.8-Flash-Next optimized for 1x RTX Pro 6000☆19Sep 17, 2026Updated last month
- English guide + one-command patcher for the vLLM spin-wait bug that overheats NVIDIA DGX Spark / GB10. Credit: nacyot. CPU cores spin at …☆24Aug 17, 2026Updated 2 months ago
- DeepSeek-V4-Flash-DSpark abliterated (uncensored) · ~100% refusal bypass · C1 ~57 tok/s · 1M ctx · 2× DGX Spark · HF weights☆54Aug 17, 2026Updated 2 months ago
- A 2MB sklearn classifier on consumer CPU beats Cohere's $10M+ 218B MoE on HumanEval☆13Updated this week
- Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audi…☆25Oct 6, 2026Updated 2 weeks ago