mosecorg / mosecLinks
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
☆860Updated last month
Alternatives and similar repositories for mosec
Users that are interested in mosec are comparing it to the libraries listed below
Sorting:
- PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.☆821Updated last month
- RayLLM - LLMs on Ray (Archived). Read README for more info.☆1,265Updated 6 months ago
- ☆412Updated last year
- Serving multiple LoRA finetuned LLM as one☆1,089Updated last year
- Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.☆644Updated last week
- MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.☆2,060Updated 3 months ago
- A high-performance inference system for large language models, designed for production environments.☆467Updated last week
- Large-scale model inference.☆632Updated 2 years ago
- LLMPerf is a library for validating and benchmarking LLMs☆1,019Updated 9 months ago
- Model Deployment at Scale on Kubernetes 🦄️☆823Updated last year
- The Triton TensorRT-LLM Backend☆891Updated this week
- ☆298Updated this week
- Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀☆1,689Updated 11 months ago
- Triton Model Navigator is an inference toolkit designed for optimizing and deploying Deep Learning models with a focus on NVIDIA GPUs.☆211Updated 5 months ago
- OpenAI compatible API for TensorRT LLM triton backend☆215Updated last year
- Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackab…☆1,584Updated last year
- Module, Model, and Tensor Serialization/Deserialization☆267Updated last month
- A Python vector database you just need - no more, no less.☆633Updated last year
- S-LoRA: Serving Thousands of Concurrent LoRA Adapters☆1,854Updated last year
- This repository contains tutorials and examples for Triton Inference Server☆778Updated last week
- A Survey of AI startups☆402Updated 2 years ago
- Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)☆275Updated last year
- Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.☆648Updated this week
- Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Serv…☆491Updated 3 weeks ago
- Fast Inference Solutions for BLOOM☆565Updated 11 months ago
- Comparison of Language Model Inference Engines☆229Updated 9 months ago
- Bagua Speeds up PyTorch☆882Updated last year
- Efficient AI Inference & Serving☆477Updated last year
- Fast, Flexible and Portable Structured Generation☆1,280Updated last week
- A throughput-oriented high-performance serving framework for LLMs☆893Updated 2 weeks ago