LLM inference decode throughput benchmark with Rich TUI dashboard. Measures token generation speed across concurrency levels and context lengths. Supports SGLang and vLLM engines.
☆129Nov 4, 2026Updated last week
Alternatives and similar repositories for llm-inference-bench
Users that are interested in llm-inference-bench are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Docker images for LLM inference (SGLang + vLLM) on NVIDIA Blackwell GPUs (SM120, CUDA 13.2)☆100Oct 20, 2026Updated 3 weeks ago
- RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink☆1,201Updated this week
- Archived: b12x development has moved to FlashInfer. See README for migration and PR-porting guidance.☆270Oct 6, 2026Updated last month
- ☆17Jun 17, 2026Updated 4 months ago
- Reproducible SGLang image for DeepSeek-V4-Flash-0731 on SM120☆38Aug 25, 2026Updated 2 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ☆800Updated this week
- NVFP4 KV cache for vLLM on SM120 (RTX PRO 6000) via FlashInfer FA2 explicit-SF-stride patch — ~1.5x fp8 pool at ~95-104% speed☆20Jun 5, 2026Updated 5 months ago
- SGLang for GLM-5.3-Flash on two or four RTX PRO 6000 Blackwell GPUs (SM120), with TP2 W4A16 and TP4 NVIDIA NVFP4 profiles.☆48Oct 11, 2026Updated last month
- Qwen3.8-Flash-Next optimized for 1x RTX Pro 6000☆20Sep 17, 2026Updated last month
- Measured LLM serving results on one DGX Station GB300, including an RTX PRO 6000 MoE expert sidecar☆15Oct 7, 2026Updated last month
- Open quantization tooling for TurboQuant-style low-bit LLM releases, stock GGUF deployment, and Apple Silicon runtime experiments.☆10May 5, 2026Updated 6 months ago
- ☆37Apr 26, 2026Updated 6 months ago
- Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, a…☆423Updated this week
- VellumForge2 is a Golang CLI for generating high-quality Direct Preference Optimization datasets via a hierarchical prompt pipeline with …☆22Feb 15, 2026Updated 8 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Profile repo — categorized index of NVFP4 model releases, DGX Spark inference stacks, Apple Silicon MLX builds, the voice-AI stack, and t…☆41Updated this week
- English guide + one-command patcher for the vLLM spin-wait bug that overheats NVIDIA DGX Spark / GB10. Credit: nacyot. CPU cores spin at …☆24Aug 17, 2026Updated 2 months ago
- Frontier MoE models on RTX PRO 6000 / RTX 5090 (sm_120): DeepSeek-V4.1-Flash on 4x RTX PRO 6000, as a GPU+CPU hybrid on 4x RTX 5090, and …☆574Updated this week
- ☆17Sep 6, 2026Updated 2 months ago
- ☆21Updated this week
- RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink☆18Updated this week
- ☆17Aug 13, 2026Updated 2 months ago
- Deploy concurrent Hermes Agent workers on unified-memory GPUs (GB10, DGX Spark) for maximum total tok/s. Profile-isolated, kanban-coordin…☆83Jul 18, 2026Updated 3 months ago
- High-performance single-GPU inference for selected model checkpoints and GPUs.☆24Oct 13, 2026Updated 3 weeks ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Give text-only LLMs vision. A tiny OpenAI-compatible proxy that lets reasoning models (DeepSeek, Qwen, GLM…) see images by querying a sep…☆49Jul 7, 2026Updated 4 months ago
- localmaxxing.com cli☆44Updated this week
- Archived fork of antirez/ds4 for GB10 DSpark — continues as Athena: github.com/xangel82/Athena-Engine☆28Oct 30, 2026Updated 2 weeks ago
- Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpar…☆414Aug 27, 2026Updated 2 months ago
- Docker compose serving stack for DeepSeek v4 Flash DSpark for NVIDIA Spark GB10 system using Aidendle94 image☆33Aug 30, 2026Updated 2 months ago
- Open-source macOS AI computer use and native GUI automation on Apple silicon. Jev + OmniParser CoreML + Apple Vision OCR. Bring your own …☆26Sep 18, 2026Updated last month
- KiChad — a Codex-integrated downstream of KiCad☆22Jul 24, 2026Updated 3 months ago
- Teach GLM-5.2 to see art: minimal frozen-backbone vision-projector training (method by Harry Partridge)☆14Jul 18, 2026Updated 3 months ago
- MiniMax-M3 (MXFP4, MSA) serving on SM120 (RTX PRO 6000 Blackwell) — patched vLLM image + launch recipe, 192K context☆22Jun 15, 2026Updated 4 months ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- 1.0 Beta · oMLX 0.6.3rc2 Dual-ANE+CPU · Qwen3.8-27B abliterated oQ4e · Mac Studio M3 Ultra☆44Aug 23, 2026Updated 2 months ago
- ☆25Sep 3, 2026Updated 2 months ago
- Concurrency Patch to enable on DSpark - DeepSeek V4 Flash/Pro C=16 can be modified for more☆12Jun 29, 2026Updated 4 months ago
- GLM-5.3-Flash (NVFP4) at TP4 across 4x DGX Spark via a switchless RoCE ring + DFlash2 — reproducible recipe☆112Oct 25, 2026Updated 2 weeks ago
- A tiered-memory system design for workloads that don't fit in RAM: measure the working set, pin the hot tier, stream the cold tier from f…☆33Nov 4, 2026Updated last week
- Unlock P2P comms between consumer NVIDIA GPUs☆719Oct 26, 2026Updated 2 weeks ago
- SparkGLM Atlas engine: Enntity's fork of Mango-kid/atlas (Atlas-Inf/atlas), AGPL-3.0☆11Updated this week
Are these results useful?