vectorch-ai / ScaleLLMLinks
A high-performance inference system for large language models, designed for production environments.
☆460Updated 3 weeks ago
Alternatives and similar repositories for ScaleLLM
Users that are interested in ScaleLLM are comparing it to the libraries listed below
Sorting:
- Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).☆246Updated last year
- A throughput-oriented high-performance serving framework for LLMs☆872Updated last week
- Efficient AI Inference & Serving☆473Updated last year
- FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.☆880Updated 11 months ago
- OpenAI compatible API for TensorRT LLM triton backend☆213Updated last year
- DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …☆263Updated 2 weeks ago
- Serving multiple LoRA finetuned LLM as one☆1,083Updated last year
- ☆289Updated 2 weeks ago
- LLM Inference benchmark☆426Updated last year
- Comparison of Language Model Inference Engines☆228Updated 8 months ago
- ☆195Updated 3 months ago
- [NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention…☆1,105Updated last week
- ☆427Updated last month
- [ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding☆1,272Updated 5 months ago
- Pretrain, finetune and serve LLMs on Intel platforms with Ray☆128Updated last week
- Accelerate inference without tears☆322Updated 5 months ago
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆309Updated this week
- RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.☆838Updated 3 weeks ago
- LLMPerf is a library for validating and benchmarking LLMs☆983Updated 8 months ago
- The Triton TensorRT-LLM Backend☆881Updated this week
- [MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Se…☆735Updated 5 months ago
- Materials for learning SGLang☆530Updated last month
- Advanced Quantization Algorithm for LLMs and VLMs, with support for CPU, Intel GPU, CUDA and HPU. Seamlessly integrated with Torchao, Tra…☆596Updated this week
- ☆560Updated last year
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆739Updated 10 months ago
- GPTQ inference Triton kernel☆305Updated 2 years ago
- ☆483Updated 2 weeks ago
- InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencie…☆404Updated this week
- ☆128Updated 7 months ago
- An innovative library for efficient LLM inference via low-bit quantization☆349Updated 11 months ago