omni-ai-npu / omni-inferLinks
Omni_Infer is a suite of inference accelerators designed for the Ascend NPU platform, offering native support and an expanding feature set.
☆71Updated last week
Alternatives and similar repositories for omni-infer
Users that are interested in omni-infer are comparing it to the libraries listed below
Sorting:
- DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …☆264Updated last month
- FlagScale is a large model toolkit based on open-sourced projects.☆353Updated last week
- ☆497Updated last week
- CPM.cu is a lightweight, high-performance CUDA implementation for LLMs, optimized for end-device inference and featuring cutting-edge tec…☆192Updated this week
- Compare different hardware platforms via the Roofline Model for LLM inference tasks.☆112Updated last year
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆56Updated 10 months ago
- ☆428Updated this week
- Materials for learning SGLang☆579Updated 3 weeks ago
- Train speculative decoding models effortlessly and port them smoothly to SGLang serving.☆400Updated this week
- RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.☆851Updated last week
- Disaggregated serving system for Large Language Models (LLMs).☆691Updated 5 months ago
- GLake: optimizing GPU memory management and IO transmission.☆480Updated 5 months ago
- 青稞Talk☆145Updated this week
- Triton Documentation in Chinese Simplified / Triton 中文文档☆82Updated 5 months ago
- ☆131Updated 3 months ago
- Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).☆248Updated last year
- SGLang kernel library for NPU☆57Updated this week
- ☆425Updated last month
- ☆70Updated 10 months ago
- LLM Inference benchmark☆426Updated last year
- KV cache store for distributed LLM inference☆335Updated last week
- [DAC'25] Official implement of "HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference"☆70Updated 3 months ago
- Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline mod…☆555Updated last year
- Efficient and easy multi-instance LLM serving☆487Updated 2 weeks ago
- A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.☆108Updated 4 months ago
- InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencie…☆407Updated last month
- A flexible and efficient training framework for large-scale alignment tasks☆425Updated this week
- ☆291Updated 2 weeks ago
- A fast communication-overlapping library for tensor/expert parallelism on GPUs.☆1,114Updated 3 weeks ago
- 一种任务级GPU算力分时调度的高性能深度学习训练平台☆696Updated last year