This repository is inspired by nano-vllm. Thanks! The core improvements include a CUDA-based Add+RMSNorm fused kernel, Chunked Prefill for a mixed scheduler, and FP8 KV Cache quantization achieved by rewriting the FlashAttention/PagedAttention Decode kernels with Triton. Hope this project helps you dive deeper into LLM inference mechanics!
☆36Oct 30, 2026Updated 2 weeks ago
Alternatives and similar repositories for nano-vllm-lite
Users that are interested in nano-vllm-lite are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆19Aug 17, 2026Updated 2 months ago
- PAROAttn based on SAGEAttn☆11Nov 30, 2025Updated 11 months ago
- TinyWebServer一百小时☆13Mar 30, 2024Updated 2 years ago
- 2nd place solution for RecSys Challenge 2025 by Synerise☆19Jul 9, 2025Updated last year
- 开箱即用的轻量级推理引擎，全面超越nano-vllm；支持Qwen3系列模型 ；流式调用与openai兼容接口/ A lightweight LLM inference engine built from scratch☆19Sep 10, 2026Updated 2 months ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- 中文计算机课程实验文库：课程实验文档、源码、环境与学习路线☆18Oct 4, 2026Updated last month
- Reimplementation of paper EIE by Stanford Song Han☆10Mar 18, 2021Updated 5 years ago
- ☆19May 28, 2024Updated 2 years ago
- Matrix Accelerator Generator for GeMM Operations based on SIGMA Architecture in CHISEL HDL☆15Mar 21, 2024Updated 2 years ago
- 1st place solution by team rec2 for RecSys Challenge 2025.☆21Jul 11, 2025Updated last year
- Official implementation of "Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models". (ECCV 2026 Spotlig…☆11Jul 3, 2026Updated 4 months ago
- Code for reproducing results for SOSP paper Bagpipe☆12Oct 20, 2023Updated 3 years ago
- ☆13Sep 14, 2026Updated last month
- ☆13May 28, 2026Updated 5 months ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- A lightweight `vLLM-Omni`-style diffusion implementation built around `Wan2.2-TI2V-5B-Diffusers` inspired from nano-vllm☆66May 25, 2026Updated 5 months ago
- SGEMM optimization with cuda step by step☆24Mar 23, 2024Updated 2 years ago
- ☆10Jul 18, 2023Updated 3 years ago
- mit-6.824 distributed system labs demo in golang & python☆11Nov 20, 2023Updated 2 years ago
- Skills to guide Claude Code, Codex, and other coding agents on using the SwanLab AI training experiment tracking platform.☆18Updated this week
- Lab for Digital Design and Computer Architecture Spring 2022 (252-0028-00L) (ETH).☆14Mar 1, 2023Updated 3 years ago
- HSTU-BLaIR: Lightweight Contrastive Text Embedding for Generative Recommender 🌱☆28Jul 4, 2025Updated last year
- ☆11Apr 29, 2023Updated 3 years ago
- 学习muduo库☆14Oct 4, 2022Updated 4 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- ☆29Jan 5, 2025Updated last year
- 北京大学未名超算队与北京大学学生 Linux 俱乐部合办的暑期 AI Infra 系列活动仓库☆256Aug 22, 2026Updated 2 months ago
- Here are some implementations of basic hardware units in RTL language (verilog for now), which can be used for area/power evaluation and …☆17Aug 25, 2023Updated 3 years ago
- 📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Py…☆28Updated this week
- Single Cycle and Pipeline CPU of RISC-V Architecture designed for Digital Design and Computer Organization Experiments 2021, NJU☆14Jan 17, 2022Updated 4 years ago
- HD-MoE: Hybrid and Dynamic Parallelism for Mixture-of-Expert LLMs with 3D Near-Memory Processing☆15Sep 2, 2026Updated 2 months ago
- A lightweight key-value storage engine based on the Bitcask model.☆40Mar 2, 2025Updated last year
- NSCSCC 2023 The Second Prize. TEAM PUA FROM HDU.☆15Mar 29, 2025Updated last year
- Accelerator RTL inspired by VEGETA [HPCA'23] and MicroScopiQ [ISCA'25]☆17Nov 11, 2025Updated last year
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- 知乎 AI 蓝宝书 · 三册文稿（AI Infra / RL / 具身智能）☆127Sep 16, 2026Updated last month
- Agent Skill for PyTRIO — teach AI coding agents to write correct remote LLM training & inference code☆26Nov 5, 2026Updated last week
- Artifact for Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization☆20May 9, 2025Updated last year
- Agent skill (Claude Code / Codex) for condensing, typesetting and fact-checking a LaTeX paper to ICASSP's 4+1 page limit, verified by mea…☆19Sep 14, 2026Updated last month
- ETH Computer Architecture - Fall 2020☆13Feb 26, 2021Updated 5 years ago
- 这是一个用chisel实现的FP8混合乘加单元☆19Jun 18, 2025Updated last year
- Research Harness for Claude Code. Keep your agent grounded in context, not lost in vibe coding.☆25Sep 5, 2026Updated 2 months ago
Are these results useful?