vectorch-ai / ScaleLLM
A high-performance inference system for large language models, designed for production environments.
☆425Updated this week
Alternatives and similar repositories for ScaleLLM:
Users that are interested in ScaleLLM are comparing it to the libraries listed below
- Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).☆238Updated last year
- A throughput-oriented high-performance serving framework for LLMs☆773Updated 6 months ago
- FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.☆771Updated 6 months ago
- ☆181Updated 5 months ago
- A flexible and efficient training framework for large-scale alignment tasks☆333Updated last month
- ☆238Updated this week
- Comparison of Language Model Inference Engines☆208Updated 3 months ago
- Efficient AI Inference & Serving☆468Updated last year
- [NeurIPS'24 Spotlight, ICLR'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which r…☆944Updated last month
- DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …☆238Updated 2 weeks ago
- RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.☆670Updated 2 months ago
- Serving multiple LoRA finetuned LLM as one☆1,040Updated 10 months ago
- OpenAI compatible API for TensorRT LLM triton backend☆201Updated 7 months ago
- [MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Se…☆611Updated 2 weeks ago
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆705Updated 5 months ago
- scalable and robust tree-based speculative decoding algorithm☆339Updated last month
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆289Updated last month
- USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference☆452Updated this week
- LLM Inference benchmark☆404Updated 8 months ago
- Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs☆224Updated this week
- Materials for learning SGLang☆345Updated this week
- InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencie…☆368Updated this week
- Official repository for LongChat and LongEval☆516Updated 10 months ago
- Easy and Efficient Quantization for Transformers☆193Updated last month
- A high-throughput and memory-efficient inference and serving engine for LLMs☆130Updated 9 months ago
- For releasing code related to compression methods for transformers, accompanying our publications☆420Updated 2 months ago
- Advanced Quantization Algorithm for LLMs/VLMs.☆403Updated this week
- ☆116Updated last year
- LLMPerf is a library for validating and benchmarking LLMs☆826Updated 3 months ago
- Dynamic Memory Management for Serving LLMs without PagedAttention☆317Updated this week