triton-inference-server / stateful_backendLinks
Triton backend for managing the model state tensors automatically in sequence batcher
☆18Updated last year
Alternatives and similar repositories for stateful_backend
Users that are interested in stateful_backend are comparing it to the libraries listed below
Sorting:
- The Triton backend for the ONNX Runtime.☆159Updated this week
- TRITONCACHE implementation of a Redis cache☆14Updated 3 weeks ago
- Triton Model Navigator is an inference toolkit designed for optimizing and deploying Deep Learning models with a focus on NVIDIA GPUs.☆210Updated 4 months ago
- Optimus is a flexible and scalable framework built to train language models efficiently across diverse hardware configurations, including…☆66Updated last month
- Tutorial on how to convert machine learned models into ONNX☆16Updated 2 years ago
- Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.☆103Updated last year
- FIL backend for the Triton Inference Server☆82Updated 2 weeks ago
- The Triton backend for TensorFlow.☆54Updated 2 months ago
- The Triton backend for the PyTorch TorchScript models.☆158Updated 3 weeks ago
- 🛠️ Tools for Transformers compression using PyTorch Lightning ⚡☆84Updated 9 months ago
- ☆68Updated 2 years ago
- experiments with inference on llama☆104Updated last year
- Triton CLI is an open source command line interface that enables users to create, deploy, and profile models served by the Triton Inferen…☆67Updated 3 weeks ago
- 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.☆137Updated last year
- Common source, scripts and utilities for creating Triton backends.☆345Updated 3 weeks ago
- Unofficial golang package for the Triton Inference Server(https://github.com/triton-inference-server/server)☆50Updated this week
- vLLM adapter for a TGIS-compatible gRPC server.☆38Updated this week
- minimal pytorch implementation of bm25 (with sparse tensors)☆104Updated last year
- OpenAI compatible API for TensorRT LLM triton backend☆214Updated last year
- Common source, scripts and utilities shared across all Triton repositories.☆76Updated 3 weeks ago
- The Triton backend for TensorRT.☆78Updated 3 weeks ago
- Benchmark suite for LLMs from Fireworks.ai☆80Updated this week
- Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)☆28Updated 2 years ago
- Make triton easier☆47Updated last year
- Various transformers for FSDP research☆38Updated 2 years ago
- MLFlow Deployment Plugin for Ray Serve☆46Updated 3 years ago
- ☆291Updated 3 weeks ago
- IBM development fork of https://github.com/huggingface/text-generation-inference☆61Updated 3 months ago
- Vector Database with support for late interaction and token level embeddings.☆55Updated 2 months ago
- Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and…☆31Updated 5 months ago