vllm-project / ci-infraLinks
This repo hosts code for vLLM CI & Performance Benchmark infrastructure.
☆23Updated this week
Alternatives and similar repositories for ci-infra
Users that are interested in ci-infra are comparing it to the libraries listed below
Sorting:
- vLLM adapter for a TGIS-compatible gRPC server.☆42Updated this week
- Pytorch DTensor native training library for LLMs/VLMs with OOTB Hugging Face support☆141Updated this week
- Common recipes to run vLLM☆196Updated this week
- ☆262Updated 2 weeks ago
- LM engine is a library for pretraining/finetuning LLMs☆74Updated this week
- Benchmark suite for LLMs from Fireworks.ai☆83Updated 2 weeks ago
- ArcticInference: vLLM plugin for high-throughput, low-latency inference☆288Updated this week
- IBM development fork of https://github.com/huggingface/text-generation-inference☆61Updated last month
- Pretrain, finetune and serve LLMs on Intel platforms with Ray☆132Updated last month
- A collection of all available inference solutions for the LLMs☆91Updated 8 months ago
- torchcomms: a modern PyTorch communications API☆219Updated this week
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆130Updated 10 months ago
- vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs☆92Updated this week
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆60Updated last year
- A collection of reproducible inference engine benchmarks☆37Updated 6 months ago
- A tool to configure, launch and manage your machine learning experiments.☆203Updated this week
- Large Language Model Text Generation Inference on Habana Gaudi☆34Updated 7 months ago
- Simple and efficient DeepSeek V3 SFT using pipeline parallel and expert parallel, with both FP8 and BF16 trainings☆88Updated 3 months ago
- ☆40Updated this week
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆344Updated 6 months ago
- Accelerating your LLM training to full speed! Made with ❤️ by ServiceNow Research☆255Updated this week
- ☆78Updated 2 months ago
- Write a fast kernel and run it on Discord. See how you compare against the best!☆58Updated 3 weeks ago
- KV cache compression for high-throughput LLM inference☆143Updated 8 months ago
- Google TPU optimizations for transformers models☆121Updated 9 months ago
- A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.☆75Updated last year
- Pipeline parallelism for the minimalist☆35Updated 2 months ago
- Load compute kernels from the Hub☆308Updated last week
- vLLM performance dashboard☆37Updated last year
- Docker image NVIDIA GH200 machines - optimized for vllm serving and hf trainer finetuning☆50Updated 8 months ago