npuichigo / openai_trtllm
OpenAI compatible API for TensorRT LLM triton backend
☆201Updated 7 months ago
Alternatives and similar repositories for openai_trtllm:
Users that are interested in openai_trtllm are comparing it to the libraries listed below
- ☆238Updated this week
- Comparison of Language Model Inference Engines☆208Updated 3 months ago
- ☆181Updated 5 months ago
- The Triton TensorRT-LLM Backend☆809Updated last week
- Easy and Efficient Quantization for Transformers☆193Updated last month
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆289Updated last month
- FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.☆771Updated 6 months ago
- A throughput-oriented high-performance serving framework for LLMs☆773Updated 6 months ago
- Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs☆224Updated this week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆262Updated 5 months ago
- A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ/VPTQ, and export to onnx/onnx-runtime easily.☆163Updated 2 weeks ago
- 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.☆136Updated 8 months ago
- ☆49Updated 4 months ago
- A high-performance inference system for large language models, designed for production environments.☆425Updated this week
- ☆116Updated last year
- Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).☆238Updated last year
- Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.☆94Updated 7 months ago
- Common source, scripts and utilities for creating Triton backends.☆310Updated last week
- LLMPerf is a library for validating and benchmarking LLMs☆826Updated 3 months ago
- Benchmark suite for LLMs from Fireworks.ai☆69Updated last month
- ☆51Updated this week
- Official implementation of Half-Quadratic Quantization (HQQ)☆770Updated this week
- Advanced Quantization Algorithm for LLMs/VLMs.☆403Updated this week
- experiments with inference on llama☆104Updated 9 months ago
- Triton Model Navigator is an inference toolkit designed for optimizing and deploying Deep Learning models with a focus on NVIDIA GPUs.☆198Updated 2 months ago
- GPTQ inference Triton kernel☆298Updated last year
- A high-throughput and memory-efficient inference and serving engine for LLMs☆130Updated 9 months ago
- Redis for LLMs☆624Updated this week
- vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs☆87Updated this week
- The Triton backend for the ONNX Runtime.☆140Updated last week