vllm混合推理扩展插件，支持多NUMA混合推理，单卡推理Qwen3-Next模型可达1000+ prefill
☆34Nov 7, 2025Updated 11 months ago
Alternatives and similar repositories for exvllm
Users that are interested in exvllm are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features …☆471Sep 22, 2026Updated last month
- The complete NUMA-optimized branch of the ktransformers project☆26Nov 3, 2025Updated 11 months ago
- Lsglang is a special extension of sglang that fully utilizes CPU and GPU computing resources with an efficient GPU parallel + NUMA parall…☆155Updated this week
- DETR tensor去除推理过程无用辅助头+fp16部署再次加速+解决转tensorrt 输出全为0问题的新方法。☆12Jan 9, 2024Updated 2 years ago
- A cmpunlocker fork with working p2p on SYS-4028GR-TRT2☆26Aug 8, 2026Updated 2 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- KTransformers 一键部署脚本☆60Apr 18, 2025Updated last year
- Local LLM Inference Speed Test Tool☆243Updated this week
- ☆14Jan 14, 2023Updated 3 years ago
- ☆14Feb 3, 2022Updated 4 years ago
- 一个mirai的群内飞行棋☆11May 12, 2023Updated 3 years ago
- ☆974Sep 14, 2025Updated last year
- FORK of VLLM for AMD MI25/50/60. A high-throughput and memory-efficient inference and serving engine for LLMs☆70May 4, 2025Updated last year
- ☆19Aug 23, 2022Updated 4 years ago
- yolo-pose for training escalator data☆16Jul 1, 2024Updated 2 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Frequency-based KV cache pruning for llama.cpp — 25% cache reduction, improved PPL at long context. GPU compaction kernel for HIP/ROCm.☆16Apr 18, 2026Updated 6 months ago
- run DeepSeek-R1 GGUFs on KTransformers☆258Mar 3, 2025Updated last year
- Implementation of FlashAttention-2 for Nvidia Tesla V100 / Titan V☆214Oct 14, 2026Updated last week
- A ROS 1/ROS 2 hybrid package wrapping the Apache TVM project.☆10Jan 24, 2023Updated 3 years ago
- TensorRT-FastSAM(https://github.com/CASIA-IVA-Lab/FastSAM)☆23Feb 29, 2024Updated 2 years ago
- 大语言模型工具集☆28Aug 1, 2025Updated last year
- This project is the Torch implementation of our ICCV 2017 paper: Centered Weight Normalization in Accelerating Training of Deep Neural…☆21Dec 7, 2019Updated 6 years ago
- fastllm是后端无依赖的高性能大模型推理库。同时支持张量并行推理稠密模型和混合模式推理MOE模型，任意10G以上显卡即可推理满血DeepSeek。双路9004/9005服务器+单显卡部署DeepSeek满血满精度原版模型，单并发20tps；INT4量化模型单并发30tp…☆5,114Updated this week
- ragflow中的ocr部分，非官方项目☆54Aug 26, 2024Updated 2 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Vstream - Video Analytics pipeline with Hardware based accelerations (dev - stage)☆10Feb 2, 2024Updated 2 years ago
- This project is intended to build and deploy an SNPE model on Qualcomm Devices, which are having unsupported layers which are not part of…☆10Oct 4, 2021Updated 5 years ago
- ☆10Feb 18, 2024Updated 2 years ago
- FlashRT-HF-kernels contains standalone FlashRT CUDA/CUTLASS kernels prepared for the Hugging Face kernels community, focusing on small-ba…☆24Aug 21, 2026Updated 2 months ago
- Converts CLIP models to ONNX☆11Jan 17, 2023Updated 3 years ago
- Excel RTD server sourcing data from Redis☆11Dec 11, 2024Updated last year
- Revision of official yolov7-pose to support custom dataset for keypoint detection☆11Nov 12, 2023Updated 2 years ago
- A C++ implementation for UCMCTrack (SOTA in MOT17)☆27May 30, 2025Updated last year
- LDC: Lightweight Dense CNN for Edge DetectionのPythonでのONNX推論サンプル☆15May 6, 2023Updated 3 years ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- YOLOv12 TensorRT 端到端模型加速推理和INT8量化实现☆14Mar 5, 2025Updated last year
- CrewAI新版本支 持使用Knowledge属性将txt、PDF、CSV、Excel、JSON等多种数据格式内容及多文件混合作为知识增强知识库提供给Crew中的Agent使用☆15Nov 27, 2024Updated last year
- Simple .NET Dynamic Data Exchange (DDE) Server☆10Aug 1, 2017Updated 9 years ago
- code☆13Jan 24, 2021Updated 5 years ago
- Kevacoin source tree☆37Feb 28, 2025Updated last year
- An onnx-based quantitation tool.☆71Jan 8, 2024Updated 2 years ago
- llama.cpp fork with additional SOTA quants and improved performance☆3,321Updated this week