sgl-project / sglang
SGLang is a fast serving framework for large language models and vision language models.
☆6,127Updated this week
Related projects ⓘ
Alternatives and complementary repositories for sglang
- LMDeploy is a toolkit for compressing, deploying, and serving LLMs.☆4,669Updated this week
- Tools for merging pretrained large language models.☆4,816Updated 2 weeks ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆30,423Updated this week
- An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.☆4,497Updated last month
- A fast inference library for running LLMs locally on modern consumer-class GPUs☆3,680Updated this week
- Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs☆2,205Updated this week
- LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalabili…☆2,613Updated this week
- Go ahead and axolotl questions☆7,930Updated this week
- A framework for few-shot evaluation of language models.☆6,990Updated this week
- Large Language Model Text Generation Inference☆9,122Updated this week
- AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:☆1,765Updated this week
- Accessible large language models via k-bit quantization for PyTorch.☆6,299Updated this week
- [MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration☆2,526Updated last month
- TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain…☆8,681Updated last week
- S-LoRA: Serving Thousands of Concurrent LoRA Adapters☆1,755Updated 9 months ago
- PyTorch native finetuning library☆4,336Updated this week
- [EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which ach…☆4,653Updated this week
- A blazing fast inference solution for text embeddings models☆2,846Updated 2 weeks ago
- Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.☆5,669Updated last month
- MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.☆1,904Updated this week
- Structured Text Generation☆9,487Updated this week
- High-speed Large Language Model Serving on PCs with Consumer-grade GPUs☆7,965Updated 2 months ago
- Supercharge Your LLM Application Evaluations 🚀☆7,261Updated this week
- The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.☆7,919Updated 6 months ago
- QLoRA: Efficient Finetuning of Quantized LLMs☆10,059Updated 5 months ago
- Train transformer language models with reinforcement learning.☆10,086Updated this week
- ☆4,035Updated 5 months ago
- 📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batc…☆2,845Updated this week
- Robust recipes to align language models with human and AI preferences☆4,680Updated last month
- Retrieval and Retrieval-augmented LLMs☆7,613Updated this week