OpenCSGs / llm-inferenceLinks
llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more.
☆89Updated last year
Alternatives and similar repositories for llm-inference
Users that are interested in llm-inference are comparing it to the libraries listed below
Sorting:
- The framework of training large language models，support lora, full parameters fine tune etc, define yaml to start training/fine tune of y…☆31Updated last year
- ☆113Updated last year
- 配合 HAI Platform 使用的集成化用户界面☆53Updated 2 years ago
- bisheng-unstructured library☆56Updated 6 months ago
- This repository provides installation scripts and configuration files for deploying the CSGHub instance, includes Helm charts and Docker…☆17Updated last week
- A Toolkit for Running On-device Large Language Models (LLMs) in APP☆79Updated last year
- [ACL2025 demo track] ROGRAG: A Robustly Optimized GraphRAG Framework☆185Updated 3 weeks ago
- XVERSE-MoE-A4.2B: A multilingual large language model developed by XVERSE Technology Inc.☆39Updated last year
- GLM Series Edge Models☆153Updated 5 months ago
- LLM scheduler user interface☆21Updated last year
- Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).☆249Updated last year
- Byzer-retrieval is a distributed retrieval system which designed as a backend for LLM RAG (Retrieval Augmented Generation). The system su…☆49Updated 8 months ago
- 国产加速卡-海光DCU实战（大模型训练、微调、推理 等）☆56Updated 3 months ago
- DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …☆267Updated 3 months ago
- xllamacpp - a Python wrapper of llama.cpp☆66Updated last week
- Easy, fast, and cheap pretrain,finetune, serving for everyone☆316Updated 4 months ago
- ☆29Updated last year
- CPM.cu is a lightweight, high-performance CUDA implementation for LLMs, optimized for end-device inference and featuring cutting-edge tec…☆205Updated last month
- App-Controller: Allow users to manipulate your App with natural language☆131Updated last year
- This is an NVIDIA AI Workbench example project that demonstrates an end-to-end model development workflow using Llamafactory.☆68Updated last year
- A benchmarking tool for comparing different LLM API providers' DeepSeek model deployments.☆30Updated 8 months ago
- Akcio is a demonstration project for Retrieval Augmented Generation (RAG). It leverages the power of LLM to generate responses and uses v…☆258Updated 2 years ago
- A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.☆79Updated last year
- vLLM Router☆51Updated last year
- Delta-CoMe can achieve near loss-less 1-bit compressin which has been accepted by NeurIPS 2024☆59Updated last year
- vLLM Documentation in Chinese Simplified / vLLM 中文文档☆128Updated last week
- LLM 推理服务性能测试☆44Updated last year
- Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.☆212Updated this week
- Its an open source LLM based on MOE Structure.☆58Updated last year
- 我们是第一个完全可商用的角色大模型。☆40Updated last year