run-ai / runai-model-streamerLinks
☆238Updated last week
Alternatives and similar repositories for runai-model-streamer
Users that are interested in runai-model-streamer are comparing it to the libraries listed below
Sorting:
- Module, Model, and Tensor Serialization/Deserialization☆256Updated this week
- Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs☆491Updated last week
- Pretrain, finetune and serve LLMs on Intel platforms with Ray☆128Updated last week
- OME is a Kubernetes operator for enterprise-grade management and serving of Large Language Models (LLMs)☆226Updated this week
- CUDA checkpoint and restore utility☆360Updated 6 months ago
- GPUd automates monitoring, diagnostics, and issue identification for GPUs☆413Updated this week
- Inference server benchmarking tool☆93Updated 3 months ago
- ☆314Updated last year
- A high-throughput and memory-efficient inference and serving engine for LLMs☆266Updated 10 months ago
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆383Updated last week
- Benchmark suite for LLMs from Fireworks.ai☆79Updated 3 weeks ago
- ☆289Updated 2 weeks ago
- IBM development fork of https://github.com/huggingface/text-generation-inference☆61Updated 3 months ago
- ☆38Updated 2 weeks ago
- Common recipes to run vLLM☆102Updated this week
- A tool to configure, launch and manage your machine learning experiments.☆182Updated this week
- High-performance safetensors model loader☆53Updated last month
- ☆55Updated 9 months ago
- ArcticInference: vLLM plugin for high-throughput, low-latency inference☆210Updated last week
- OpenAI compatible API for TensorRT LLM triton backend☆213Updated last year
- JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs wel…☆369Updated 2 months ago
- Helm charts for llm-d☆50Updated last month
- xet client tech, used in huggingface_hub☆171Updated this week
- ☆58Updated 11 months ago
- A Lossless Compression Library for AI pipelines☆274Updated last month
- An Operator for deployment and maintenance of NVIDIA NIMs and NeMo microservices in a Kubernetes environment.☆124Updated this week
- KAI Scheduler is an open source Kubernetes Native scheduler for AI workloads at large scale☆756Updated this week
- A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.☆70Updated last year
- Distributed Model Serving Framework☆174Updated 2 months ago
- ☆217Updated 7 months ago