nickaggarwal / nvidia-triton-llm-streaming
Integrating SSE with NVIDIA Triton Inference Server using a Python backend and Zephyr model. There is very less documentation how to use Nvidia Triton in Streaming use-cases ( hard to find in their docs ), hence this should be helpful for people who want to deploy streaming with Triton
☆10Updated 9 months ago
Alternatives and similar repositories for nvidia-triton-llm-streaming:
Users that are interested in nvidia-triton-llm-streaming are comparing it to the libraries listed below
- ☆238Updated this week
- OpenAI compatible API for TensorRT LLM triton backend☆201Updated 7 months ago
- Triton backend for https://github.com/OpenNMT/CTranslate2☆34Updated last year
- The Triton backend for the ONNX Runtime.☆140Updated last week
- This repository contains tutorials and examples for Triton Inference Server☆671Updated this week
- The Triton TensorRT-LLM Backend☆809Updated last week
- Triton Model Navigator is an inference toolkit designed for optimizing and deploying Deep Learning models with a focus on NVIDIA GPUs.☆199Updated 2 months ago
- Easy and Efficient Quantization for Transformers☆195Updated last month
- Triton CLI is an open source command line interface that enables users to create, deploy, and profile models served by the Triton Inferen…☆60Updated last week
- Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Serv…☆462Updated 2 weeks ago
- 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.☆136Updated 8 months ago
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆289Updated last month
- A high-throughput and memory-efficient inference and serving engine for LLMs☆262Updated 5 months ago
- The Triton backend for TensorRT.☆70Updated 2 weeks ago
- ☆32Updated 2 years ago
- Common source, scripts and utilities for creating Triton backends.☆311Updated this week
- PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.☆781Updated last month
- The Triton backend for the PyTorch TorchScript models.☆144Updated 2 weeks ago
- Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.☆597Updated this week
- ☆181Updated 5 months ago
- ☆51Updated this week
- A unified library of state-of-the-art model optimization techniques such as quantization, pruning, distillation, speculative decoding, et…☆821Updated last week
- NVIDIA Riva runnable tutorials☆127Updated last week
- A toolkit for processing speech data and creating speech datasets☆106Updated last week
- A tool to configure, launch and manage your machine learning experiments.☆129Updated this week
- Triton backend for https://github.com/OpenNMT/CTranslate2☆11Updated 7 months ago
- Comparison of Language Model Inference Engines☆208Updated 3 months ago
- An innovative library for efficient LLM inference via low-bit quantization☆351Updated 6 months ago
- Whisper inference with TensorRT-LLM☆21Updated last year
- Google TPU optimizations for transformers models☆104Updated 2 months ago