CactusQ / TensorRT-LLM-TutorialLinks
Getting started with TensorRT-LLM using BLOOM as a case study
☆20Updated last year
Alternatives and similar repositories for TensorRT-LLM-Tutorial
Users that are interested in TensorRT-LLM-Tutorial are comparing it to the libraries listed below
Sorting:
- Notes on quantization in neural networks☆96Updated last year
- A family of compressed models obtained via pruning and knowledge distillation☆347Updated 8 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆266Updated 10 months ago
- ☆286Updated last week
- Code for HyperSeg and HyperSum☆15Updated 3 weeks ago
- This repository contains tutorials and examples for Triton Inference Server☆751Updated last week
- ☆187Updated 7 months ago
- A collection of all available inference solutions for the LLMs☆91Updated 5 months ago
- ☆148Updated this week
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆326Updated 3 months ago
- This reference can be used with any existing OpenAI integrated apps to run with TRT-LLM inference locally on GeForce GPU on Windows inste…☆126Updated last year
- This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT)…☆70Updated last year
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆308Updated 2 months ago
- Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector…☆305Updated 9 months ago
- A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ/VPTQ, and export to onnx/onnx-runtime easily.☆175Updated 4 months ago
- The Triton TensorRT-LLM Backend☆875Updated this week
- A repository dedicated to evaluating the performance of quantizied LLaMA3 using various quantization methods..☆193Updated 6 months ago
- For releasing code related to compression methods for transformers, accompanying our publications☆437Updated 6 months ago
- Efficient LLM Inference over Long Sequences☆387Updated last month
- Advanced Quantization Algorithm for LLMs and VLMs, with support for CPU, Intel GPU, CUDA and HPU. Seamlessly integrated with Torchao, Tra…☆574Updated this week
- Official implementation of Half-Quadratic Quantization (HQQ)☆856Updated last week
- An innovative library for efficient LLM inference via low-bit quantization☆349Updated 11 months ago
- Collection of reference workflows for building intelligent agents with NIMs☆167Updated 6 months ago
- A unified library of state-of-the-art model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. …☆1,103Updated this week
- Easy and Efficient Quantization for Transformers☆198Updated last month
- Integrating SSE with NVIDIA Triton Inference Server using a Python backend and Zephyr model. There is very less documentation how to use …☆10Updated last year
- Unofficial implementation of https://arxiv.org/pdf/2407.14679☆48Updated 11 months ago
- Comparison of Language Model Inference Engines☆225Updated 7 months ago
- OpenAI compatible API for TensorRT LLM triton backend☆210Updated last year
- How to quickly serve an LLM using Fast API, Celery, and Redis☆16Updated last year