opendatahub-io / vllm-tgis-adapterLinks
vLLM adapter for a TGIS-compatible gRPC server.
☆32Updated this week
Alternatives and similar repositories for vllm-tgis-adapter
Users that are interested in vllm-tgis-adapter are comparing it to the libraries listed below
Sorting:
- 👷 Build compute kernels☆68Updated this week
- Benchmark suite for LLMs from Fireworks.ai☆76Updated 2 weeks ago
- Inference server benchmarking tool☆74Updated 2 months ago
- Lightweight toolkit package to train and fine-tune 1.58bit Language models☆80Updated last month
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆116Updated 6 months ago
- Hugging Face Inference Toolkit used to serve transformers, sentence-transformers, and diffusers models.☆78Updated 2 weeks ago
- ArcticTraining is a framework designed to simplify and accelerate the post-training process for large language models (LLMs)☆130Updated this week
- vLLM performance dashboard☆30Updated last year
- Load compute kernels from the Hub☆191Updated this week
- ☆159Updated this week
- A Python wrapper around HuggingFace's TGI (text-generation-inference) and TEI (text-embedding-inference) servers.☆33Updated last month
- ☆34Updated last month
- Train, tune, and infer Bamba model☆127Updated 3 weeks ago
- ☆41Updated 2 weeks ago
- ☆53Updated last year
- ☆35Updated last year
- Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…☆55Updated this week
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆59Updated 8 months ago
- A safetensors extension to efficiently store sparse quantized tensors on disk☆129Updated this week
- ☆74Updated 7 months ago
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆53Updated 7 months ago
- IBM development fork of https://github.com/huggingface/text-generation-inference☆60Updated last month
- Repo hosting codes and materials related to speeding LLMs' inference using token merging.☆36Updated last year
- patches for huggingface transformers to save memory☆23Updated 3 weeks ago
- Data preparation code for CrystalCoder 7B LLM☆45Updated last year
- Simple high-throughput inference library☆119Updated last month
- Code for KaLM-Embedding models☆78Updated 3 months ago
- Repository for CPU Kernel Generation for LLM Inference☆26Updated last year
- Google TPU optimizations for transformers models☆113Updated 5 months ago
- Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)☆28Updated last year