SkyworkAI / vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
☆16Updated 9 months ago
Alternatives and similar repositories for vllm:
Users that are interested in vllm are comparing it to the libraries listed below
- OneFlow Serving☆20Updated 2 months ago
- ☆15Updated 11 months ago
- Transformer related optimization, including BERT, GPT☆17Updated last year
- [ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models☆22Updated 11 months ago
- ☆23Updated last year
- ☆11Updated last year
- An easy way to run, test, benchmark and tune OpenCL kernel files☆23Updated last year
- OneFlow->ONNX☆42Updated last year
- ☆72Updated 3 months ago
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆128Updated 9 months ago
- ☆18Updated last year
- A toolkit for developers to simplify the transformation of nn.Module instances. It's now corresponding to Pytorch.fx.☆13Updated last year
- Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.☆34Updated this week
- Odysseus: Playground of LLM Sequence Parallelism☆66Updated 8 months ago
- 大模型部署实战：TensorRT-LLM, Triton Inference Server, vLLM☆26Updated last year
- An object detection codebase based on MegEngine.☆28Updated 2 years ago
- NVIDIA TensorRT Hackathon 2023复赛选题：通义千问Qwen-7B用TensorRT-LLM模型搭建及优化☆41Updated last year
- Whisper in TensorRT-LLM☆15Updated last year
- ☢️ TensorRT 2023复赛——基于TensorRT-LLM的Llama模型推断加速优化☆46Updated last year
- Implementation of IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs (ICLR 2024).☆22Updated 8 months ago
- GPTQ inference TVM kernel☆39Updated 10 months ago
- IntLLaMA: A fast and light quantization solution for LLaMA☆18Updated last year
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆40Updated 4 months ago
- ☆19Updated 10 months ago
- Summary of system papers/frameworks/codes/tools on training or serving large model☆56Updated last year
- ☆23Updated last year
- Our 2nd-gen LMM☆33Updated 9 months ago
- ☆124Updated last year