omni-ai-npu / omni-inferLinks
Omni_Infer is a suite of inference accelerators designed for the Ascend NPU platform, offering native support and an expanding feature set.
☆94Updated last week
Alternatives and similar repositories for omni-infer
Users that are interested in omni-infer are comparing it to the libraries listed below
Sorting:
- FlagScale is a large model toolkit based on open-sourced projects.☆431Updated this week
- Materials for learning SGLang☆714Updated this week
- ☆518Updated last month
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆61Updated last year
- Disaggregated serving system for Large Language Models (LLMs).☆761Updated 9 months ago
- Compare different hardware platforms via the Roofline Model for LLM inference tasks.☆120Updated last year
- CPM.cu is a lightweight, high-performance CUDA implementation for LLMs, optimized for end-device inference and featuring cutting-edge tec…☆214Updated 3 months ago
- ☆161Updated last month
- Train speculative decoding models effortlessly and port them smoothly to SGLang serving.☆615Updated this week
- ☆441Updated 5 months ago
- DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …☆272Updated 5 months ago
- Efficient and easy multi-instance LLM serving☆520Updated 4 months ago
- RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.☆984Updated this week
- GLake: optimizing GPU memory management and IO transmission.☆494Updated 9 months ago
- Fast and memory-efficient exact attention☆107Updated 3 weeks ago
- Accelerate inference without tears☆371Updated last month
- [NeurIPS 2024] Efficient LLM Scheduling by Learning to Rank☆66Updated last year
- LLM Inference with Deep Learning Accelerator.☆56Updated 11 months ago
- SGLang kernel library for NPU☆91Updated this week
- FlagCX is a scalable and adaptive cross-chip communication library.☆138Updated this week
- 青稞Talk☆181Updated this week
- [DAC'25] Official implement of "HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference"☆95Updated 3 weeks ago
- ☆153Updated 10 months ago
- ☆160Updated 5 months ago
- A high-performance RL training-inference weight synchronization framework, designed to enable second-level parameter updates from trainin…☆123Updated 2 weeks ago
- DeepSeek Native Sparse Attention pytorch implementation☆111Updated 3 weeks ago
- ☆144Updated last year
- A flexible and efficient training framework for large-scale alignment tasks☆447Updated 2 months ago
- Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).☆250Updated last year
- [OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable☆206Updated last year