Everything you need to know about LLM inference
☆481Nov 12, 2026Updated this week
Alternatives and similar repositories for llm-inference-handbook
Users that are interested in llm-inference-handbook are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆12Jul 5, 2023Updated 3 years ago
- Benchmark and optimize LLM inference across frameworks with ease☆204Oct 20, 2026Updated 3 weeks ago
- API serving for your diffusers models☆11Jan 19, 2024Updated 2 years ago
- how to build a sentence embedding application using BentoML☆15Nov 9, 2026Updated last week
- ☆16Jul 5, 2025Updated last year
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Agent Skills for Mojo and MAX development☆211Updated this week
- Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, T…☆527Updated this week
- 🚀 Launching Bento in a Kubernetes cluster☆17Oct 23, 2026Updated 3 weeks ago
- 🐳 Build OCI images for Bentos in k8s☆17Oct 23, 2026Updated 3 weeks ago
- learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen☆5,742Updated this week
- ☆149Updated this week
- LMCache: Supercharge Your LLM with the Fastest KV Cache Layer☆12,367Updated this week
- VibeRL is a Reinforcement Learning framework built essentially through vibe coding with Kimi K2.☆18Updated this week
- Tiered optimizer state allocation for memory efficient MoE training. Cuts optimizer memory by 97.4%, outperforming AdamW/Muon/Lion while …☆14Sep 10, 2026Updated 2 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- A Datacenter Scale Distributed Inference Serving Framework☆8,492Updated this week
- SGLang is a high-performance serving framework for large language models and multimodal models.☆38,899Updated this week
- Build an LLM from scratch with MAX☆78Updated this week
- My personal website and blog, built with Next.js☆11Oct 24, 2026Updated 3 weeks ago
- ☆20Mar 20, 2025Updated last year
- A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.☆5,582May 17, 2026Updated 5 months ago
- Donkey, often known by its file name DONKEY.BAS, is a computer game written in 1981 by Microsoft co-founder Bill Gates and Neil Konzen. T…☆15Aug 13, 2026Updated 3 months ago
- Code, labs, and resources for O'Reilly AI Systems Performance Engineering: GPU optimization, distributed training, inference scaling, and…☆2,214Nov 3, 2026Updated last week
- [EMNLP 2024] Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning☆10Oct 11, 2024Updated 2 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- A high-throughput and memory-efficient inference and serving engine for LLMs☆96,964Updated this week
- A tutorial on modern GPU programming for machine learning systems☆1,387Nov 2, 2026Updated last week
- Model Serving Signals☆20Updated this week
- Nano vLLM☆16,438Apr 26, 2026Updated 6 months ago
- An extension of the nanoGPT repository for training small MOE models.☆293Mar 9, 2025Updated last year
- Learning notes for understanding modern Machine Learning System.☆66Updated this week
- FlashInfer: Kernel Library for LLM Serving☆6,928Updated this week
- The Little Book of Reinforcement Learning☆1,746Jul 14, 2026Updated 4 months ago
- ☆25Oct 10, 2025Updated last year
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.☆7,045Updated this week
- State of the art inference performance with modern accelerators on Kubernetes☆5,264Updated this week
- A curated list of awesome inference engineering tools, papers and resources.☆58Updated this week
- TokenSpeed is a speed-of-light LLM inference engine.☆2,929Updated this week
- CLIP as a service - Embed image and sentences, object recognition, visual reasoning, image classification and reverse image search☆67Updated this week
- The best ChatGPT that $100 can buy.☆61Updated this week
- a fun and educational take on vLLM☆232Jan 25, 2026Updated 9 months ago
Are these results useful?