Omni_Infer is a suite of inference accelerators designed for the Ascend NPU platform, offering native support and an expanding feature set.
☆133Sep 20, 2026Updated last month
Alternatives and similar repositories for omni-infer
Users that are interested in omni-infer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Community maintained hardware plugin for vLLM on Huawei Ascend☆3,039Updated this week
- SGLang kernel library for NPU☆197Updated this week
- Visualize the Expert Parallelism Load Balancer☆18Mar 15, 2025Updated last year
- Triton language and compiler for Huawei Ascend NPU☆208Updated this week
- A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Fou…☆1,614Updated this week
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Nano vLLM Triton☆14Aug 27, 2025Updated last year
- Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.☆6,953Updated this week
- Nano vLLM☆15Jun 1, 2026Updated 5 months ago
- nano-vllm是开源的一个gpu推理项目，基于开源版本弄的一个ascend npu版本推理小demo，旨在帮助初学者了解推理的整体流程，区别于vllm，nano-vllm体量更小，麻雀虽小五脏俱全，更有助于初学者学习 。☆96May 4, 2026Updated 5 months ago
- NVIDIA Inference Xfer Library (NIXL)☆1,356Updated this week
- LMCache-Ascend is a plugin for running LMCache on the Ascend NPU.☆93Updated this week
- ☆11Jan 26, 2022Updated 4 years ago
- Composable and Embeddable Communication Runtime for Distributed AI Services☆102Oct 7, 2026Updated 3 weeks ago
- ☆553Aug 27, 2026Updated 2 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- ☆65Updated this week
- 提供给Android使用的inotifywait和inotifywatch☆14Nov 7, 2022Updated 3 years ago
- FlashInfer: Kernel Library for LLM Serving☆6,840Updated this week
- KV cache store for distributed LLM inference☆451Sep 16, 2026Updated last month
- ☆28Jun 8, 2026Updated 4 months ago
- Memory-bounded compressed sparse attention via streaming top-k. Triton kernels for the DeepSeek-V4 lightning indexer. 32x regime extensio…☆27May 5, 2026Updated 5 months ago
- A throughput-oriented high-performance serving framework for LLMs☆976Mar 29, 2026Updated 7 months ago
- dInfer: An Efficient Inference Framework for Diffusion Language Models☆488Feb 11, 2026Updated 8 months ago
- ☆21Jul 28, 2026Updated 3 months ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- GLM-5.3-Flash adaptation + offload perf patches for FreeToken v0.1.2 (17.8→36.7 tok/s on one RTX PRO 6000)☆21Oct 2, 2026Updated last month
- A Triton JIT runtime and ffi provider in C++☆41Sep 24, 2026Updated last month
- Persist and reuse KV Cache to speedup your LLM.☆350Updated this week
- A Datacenter Scale Distributed Inference Serving Framework☆8,425Updated this week
- A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.☆141Dec 25, 2025Updated 10 months ago
- ☆27Updated this week
- A high-performance and light-weight router for vLLM large scale deployment☆494Updated this week
- ☆56Jul 7, 2025Updated last year
- A benchmarking tool for comparing different LLM API providers' DeepSeek model deployments.☆31Mar 28, 2025Updated last year
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- 分享AI Infra知识&代码练习：PyTorch、vLLM/SGLang、slime/vime框架入门⚡️、性能加速🚀、大模型基础🧠、AI软硬件🔧等☆4,295Sep 20, 2026Updated last month
- A high-throughput and memory-efficient inference and serving engine for LLMs☆16Jul 19, 2026Updated 3 months ago
- The code based on vLLM for the paper “ Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention”.☆11Sep 19, 2024Updated 2 years ago
- Offline optimization of your disaggregated Dynamo graph☆463Updated this week
- Next-Generation AI-Assisted Kernel Engineering for Multi-Chip Systems☆89Sep 28, 2026Updated last month
- Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels☆8,890Updated this week
- A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM☆934Updated this week