LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, supporting hybrid inference for MOE large models.
☆465Sep 22, 2026Updated 2 weeks ago
Alternatives and similar repositories for Lvllm
Users that are interested in Lvllm are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Lsglang is a special extension of sglang that fully utilizes CPU and GPU computing resources with an efficient GPU parallel + NUMA parall…☆144Oct 5, 2026Updated last week
- 大语言模型工具集☆28Aug 1, 2025Updated last year
- forked from vllm-project/flash-attention☆67May 9, 2026Updated 5 months ago
- A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations☆19,591Updated this week
- The definitive vLLM runtime for dual RTX 2080 Ti 22GB + NVLink, delivering Qwen 27B local inference with maximum 200+ tok/s single-reques…☆1,121Oct 1, 2026Updated last week
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- V100 / SM70-focused vLLM engineering fork for modern LLM inference.☆1,276Updated this week
- vLLM for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60☆435Feb 20, 2026Updated 7 months ago
- llama.cpp fork with additional SOTA quants and improved performance☆3,293Updated this week
- Unlock P2P comms between consumer NVIDIA GPUs☆565Sep 16, 2026Updated 3 weeks ago
- ☆43May 6, 2025Updated last year
- Qwen4-Exp (Qwen3.8-Flash-Next) SWA + MTP + PLE-on-iswa — bounded deep-context decode w/ long-range recall, TBQ4 4.25 bpv KV, DSpark + Rot…☆95Sep 24, 2026Updated 2 weeks ago
- DeepSeek-V4-Flash on Ampere SM 8.6 via vLLM (pyref kernel replacements)☆35Jul 2, 2026Updated 3 months ago
- An extension utility for llama.cpp, used with 3090*2 + Strix Halo. llama.cpp的拓展小工具，自用于3090*2 + Strix Halo。☆404Sep 7, 2026Updated last month
- KTransformers 一键部署脚本☆60Apr 18, 2025Updated last year
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- IMMA-based **FP8-as-storage** GEMM experiments for Ampere (sm_86 / RTX 3090 Ti).☆28Jan 30, 2026Updated 8 months ago
- LLM speculative inference server for heterogeneous hardware & consumer GPUs☆2,911Updated this week
- triton for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60☆48Dec 8, 2025Updated 10 months ago
- Flash Attention 2 implementation for Turing GPUs☆129Sep 14, 2026Updated 3 weeks ago
- SGLang is a high-performance serving framework for large language models and multimodal models.☆36,988Updated this week
- The LLM API Benchmark Tool is a flexible Go-based utility designed to measure and analyze the performance of OpenAI-compatible API endpoi…☆104Mar 25, 2026Updated 6 months ago
- LMDeploy is a toolkit for compressing, deploying, and serving LLMs.☆8,103Updated this week
- RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink☆1,131Updated this week
- Patched NVIDIA GRID Drivers☆38May 1, 2026Updated 5 months ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- FastAPI to serve Qwen-ASR with streaming support. Tested. Benchmarked. Flash Attention 2. Fast & Stable.☆16Jun 24, 2026Updated 3 months ago
- run DeepSeek-R1 GGUFs on KTransformers☆258Mar 3, 2025Updated last year
- aha model inference library, now supports Qwen(2.5VL/3/3VL/3.5/ASR/3Embedding/3Reranker), MiniCPM(4/5), VoxCPM(0.5B/1.5/2), DeepSeek-OCR/…☆392Jun 7, 2026Updated 4 months ago
- Docker images for LLM inference (SGLang + vLLM) on NVIDIA Blackwell GPUs (SM120, CUDA 13.2)☆84Updated this week
- Archived: b12x development has moved to FlashInfer. See README for migration and PR-porting guidance.☆271Updated this week
- ☆16Jul 18, 2023Updated 3 years ago
- llama.cpp fork with additional SOTA quants and improved performance☆22Oct 4, 2026Updated last week
- Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.☆334Sep 9, 2026Updated last month
- ☆78Apr 1, 2026Updated 6 months ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- Reddit daily news☆14Sep 14, 2025Updated last year
- Implementation of FlashAttention-2 for Nvidia Tesla V100 / Titan V☆211Sep 22, 2026Updated 2 weeks ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆93,564Updated this week
- Sage attention for turning.☆83Dec 29, 2025Updated 9 months ago
- A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.☆3,520Updated this week
- gpt_server是一个用于生产级部署LLMs、Embedding、Reranker、ASR、TTS、文生图、图片编辑和文生视频的开源框架。☆253May 9, 2026Updated 5 months ago
- A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.☆5,811Updated this week