A Practitioner handbook for production llm serving.
☆263Oct 21, 2026Updated this week
Alternatives and similar repositories for llm-inference-at-scale
Users that are interested in llm-inference-at-scale are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Answer-aware memory for Markdown notes: hybrid search + pluggable passage reranker, as a Claude Code plugin☆12Oct 15, 2026Updated last week
- ModSleuth builds evidence-grounded dependency graphs for LLM releases from public artifacts such as model cards, dataset cards, technical…☆17Jun 15, 2026Updated 4 months ago
- Intro to Hyperdimensional Computing (HDC) using a tea dataset☆18Sep 29, 2026Updated 3 weeks ago
- A local MCP (Model Context Protocol) server for portable, long-term agentic memory — a thumb drive for your AI context.☆21Aug 31, 2026Updated last month
- 100 days of LLM inference engineering — daily posts, experiments, and visualizations☆1,876Apr 30, 2026Updated 5 months ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Notebook - Fine-tuning Vision-Language Models (VLMs) on custom datasets☆11Feb 9, 2025Updated last year
- Dasein managed vector index service - Python SDK☆23May 11, 2026Updated 5 months ago
- Proxima lets existing GPUs serve 4x more concurrent requests☆26Aug 19, 2026Updated 2 months ago
- OpenSpec Zed extension for my fellow Rustacean sympathizers!☆15Mar 13, 2026Updated 7 months ago
- A database-agnostic Agentic GraphRAG framework using swappable System One models (local Laya / cloud Jev). A plug-and-play intelligence l…☆51Sep 29, 2026Updated 3 weeks ago
- Curated collection of AI inference engineering resources — LLM serving, GPU kernels, quantization, distributed inference, and production …☆352Feb 4, 2026Updated 8 months ago
- CrowdGPT is an opensource AI that aims to be trained without any datacenters☆56Updated this week
- ☆53Aug 1, 2026Updated 2 months ago
- Experimental Chrome side-panel browser agent with LangGraph.js, MCP/WebMCP, local models, and semantic browser automation.☆20May 18, 2026Updated 5 months ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- A 4B decision model: calibrated probabilities for yes/no, choice and score questions over long inputs.☆43Updated this week
- A curated resource list for learning AI performance engineering, from GPU fundamentals to production inference.☆4,275Updated this week
- GPU Engineering for AI Systems☆604May 17, 2026Updated 5 months ago
- ☆12Oct 25, 2023Updated 2 years ago
- Visual-first architecture workspace with typed graphs, deterministic validation, semantic diffs, and proposal-gated AI changes.☆16Aug 27, 2026Updated last month
- Developing K - a language model to generate OPENSCAD code from prompt☆19Dec 3, 2025Updated 10 months ago
- NATSSL: Zero-Configuration Distributed TLS for Private Infrastructure☆13Jun 20, 2026Updated 4 months ago
- Distributed-AI-Systems☆68Oct 10, 2026Updated last week
- Real-time terminal UI dashboard for monitoring vLLM☆33Jul 13, 2026Updated 3 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Litespeed — a local coding agent with multi-model workflows for your terminal and browser☆28Sep 24, 2026Updated 3 weeks ago
- Run open AI models on your own computer — a friendly desktop app and local gateway for llama.cpp and more. Private, free, open source.☆12Updated this week
- ARMv8A emulator that runs in a web browser via Web Assembly☆16Updated this week
- Deployment & experimentation of LLM inference at production scale (P/D disaggregation, KV offloading, parallelism, and autoscaling).☆62Sep 26, 2026Updated 3 weeks ago
- Companion code for The Physics of LLM Inference book☆30Apr 21, 2026Updated 6 months ago
- ☆10Jul 20, 2023Updated 3 years ago
- NVFP4 KV cache for vLLM on SM120 (RTX PRO 6000) via FlashInfer FA2 explicit-SF-stride patch — ~1.5x fp8 pool at ~95-104% speed☆20Jun 5, 2026Updated 4 months ago
- Pi compaction that reuses the server's prefix cache (vLLM / SGLang / llama.cpp Anthropic endpoints): no cold re-prefill, plus a warm-up s…☆20Oct 16, 2026Updated last week
- Self-host Supabase in production, AI-ready: read-only MCP for your coding agent with zero secret exposure, SSO, PITR backups, monitoring,…☆24Updated this week
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Learn harness engineering by building Claude Code from scratch. Free, open-source course: 8 articles, 6 videos, 1 codebase.☆683Updated this week
- A even lighterweight subagent extension for the pi coding agent☆22Jul 24, 2026Updated 2 months ago
- A dbt package to run natural language queries☆10Jan 13, 2023Updated 3 years ago
- A modular, multilingual, and multimodal Retrieval-Augmented Generation (RAG) system tailored for the financial analysis of Public Investm…☆14Oct 14, 2025Updated last year
- code for training and using chess embeddings models☆14Jun 9, 2024Updated 2 years ago
- GitAra 🦜🎸: A small function-calling git agent you can run locally☆30Jul 22, 2026Updated 3 months ago
- A machine-enforced workflow protocol for AI coding agents that eliminates false progress. Tasks require verifiable evidence to close, whi…☆30Updated this week