asprenger / ray_vllm_inferenceLinks
A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
☆73Updated last year
Alternatives and similar repositories for ray_vllm_inference
Users that are interested in ray_vllm_inference are comparing it to the libraries listed below
Sorting:
- Pretrain, finetune and serve LLMs on Intel platforms with Ray☆131Updated last month
- vLLM Router☆44Updated last year
- OpenAI compatible API for TensorRT LLM triton backend☆215Updated last year
- ☆64Updated 6 months ago
- ☆302Updated this week
- Self-host LLMs with vLLM and BentoML☆152Updated 2 weeks ago
- ☆56Updated 11 months ago
- A collection of all available inference solutions for the LLMs☆91Updated 7 months ago
- Comparison of Language Model Inference Engines☆231Updated 10 months ago
- 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.☆139Updated last year
- Deployment a light and full OpenAI API for production with vLLM to support /v1/embeddings with all embeddings models.☆44Updated last year
- Benchmarking the serving capabilities of vLLM☆54Updated last year
- Benchmark suite for LLMs from Fireworks.ai☆82Updated last week
- A high-performance inference system for large language models, designed for production environments.☆479Updated 2 weeks ago
- A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM☆60Updated this week
- ☆257Updated last week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆266Updated last year
- vLLM adapter for a TGIS-compatible gRPC server.☆41Updated last week
- Accelerating your LLM training to full speed! Made with ❤️ by ServiceNow Research☆252Updated last week
- Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs☆655Updated this week
- ☆51Updated last year
- ☆82Updated 10 months ago
- A library integrating embedding and reranker models from OpenAI, SentenceTransformers etc for semantic search in vector database.☆57Updated 6 months ago
- TextEmbed is a REST API crafted for high-throughput and low-latency embedding inference. It accommodates a wide variety of embedding mode…☆25Updated last year
- This is an NVIDIA AI Workbench example project that demonstrates an end-to-end model development workflow using Llamafactory.☆67Updated last year
- Data preparation code for Amber 7B LLM☆92Updated last year
- A pipeline for LLM knowledge distillation☆109Updated 6 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆131Updated last year
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆58Updated 11 months ago
- ArcticInference: vLLM plugin for high-throughput, low-latency inference☆283Updated this week