omni-ai-npu / omni-inferLinks
Omni_Infer is a suite of inference accelerators designed for the Ascend NPU platform, offering native support and an expanding feature set.
☆60Updated this week
Alternatives and similar repositories for omni-infer
Users that are interested in omni-infer are comparing it to the libraries listed below
Sorting:
- DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …☆263Updated 3 weeks ago
- Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).☆247Updated last year
- Compare different hardware platforms via the Roofline Model for LLM inference tasks.☆112Updated last year
- CPM.cu is a lightweight, high-performance CUDA implementation for LLMs, optimized for end-device inference and featuring cutting-edge tec…☆175Updated this week
- ☆408Updated 3 weeks ago
- FlagScale is a large model toolkit based on open-sourced projects.☆347Updated last week
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆56Updated 9 months ago
- 青稞Talk☆135Updated this week
- ☆485Updated 3 weeks ago
- ☆79Updated last year
- ☆135Updated 6 months ago
- Efficient AI Inference & Serving☆476Updated last year
- Accelerate inference without tears☆322Updated 5 months ago
- Triton Documentation in Chinese Simplified / Triton 中文文档☆80Updated 4 months ago
- Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.☆89Updated this week
- LLM Inference benchmark☆426Updated last year
- An industrial extension library of pytorch to accelerate large scale model training☆47Updated 2 weeks ago
- Mixture-of-Experts (MoE) Language Model☆189Updated 11 months ago
- 中文版 llm-numbers☆123Updated last year
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆136Updated last year
- Train speculative decoding models effortlessly and port them smoothly to SGLang serving.☆344Updated this week
- A flexible and efficient training framework for large-scale alignment tasks☆416Updated this week
- 一种任务级GPU算力分时调度的高性能深度学习训练平台☆692Updated last year
- GLake: optimizing GPU memory management and IO transmission.☆478Updated 5 months ago
- Delta-CoMe can achieve near loss-less 1-bit compressin which has been accepted by NeurIPS 2024☆56Updated 9 months ago
- ☆172Updated this week
- ☆124Updated 2 months ago
- llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deploy…☆86Updated last year
- Ling is a MoE LLM provided and open-sourced by InclusionAI.☆187Updated 3 months ago
- Official implementation of ICML 2024 paper "ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking".☆48Updated last year