CactusQ / TensorRT-LLM-TutorialLinks
Getting started with TensorRT-LLM using BLOOM as a case study
☆23Updated last year
Alternatives and similar repositories for TensorRT-LLM-Tutorial
Users that are interested in TensorRT-LLM-Tutorial are comparing it to the libraries listed below
Sorting:
- A collection of all available inference solutions for the LLMs☆91Updated 8 months ago
- Integrating SSE with NVIDIA Triton Inference Server using a Python backend and Zephyr model. There is very less documentation how to use …☆10Updated last year
- This reference can be used with any existing OpenAI integrated apps to run with TRT-LLM inference locally on GeForce GPU on Windows inste…☆126Updated last year
- A family of compressed models obtained via pruning and knowledge distillation☆355Updated 11 months ago
- ☆304Updated this week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆266Updated last year
- Unofficial implementation of https://arxiv.org/pdf/2407.14679☆50Updated last year
- Comparison of Language Model Inference Engines☆233Updated 10 months ago
- ☆221Updated last month
- OpenAI compatible API for TensorRT LLM triton backend☆216Updated last year
- This repository contains tutorials and examples for Triton Inference Server☆795Updated 3 weeks ago
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆317Updated last month
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆344Updated 6 months ago
- Easy and Efficient Quantization for Transformers☆202Updated 4 months ago
- Notes on quantization in neural networks☆104Updated last year
- A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ/VPTQ, and export to onnx/onnx-runtime easily.☆180Updated 7 months ago
- This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT)…☆72Updated 2 years ago
- Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs☆667Updated this week
- ☆210Updated 10 months ago
- A repository dedicated to evaluating the performance of quantizied LLaMA3 using various quantization methods..☆195Updated 9 months ago
- Advanced Quantization Algorithm for LLMs and VLMs, with support for CPU, Intel GPU, CUDA and HPU.☆679Updated last week
- Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation☆35Updated last year
- A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM☆62Updated this week
- Efficient LLM Inference over Long Sequences☆390Updated 4 months ago
- 3x Faster Inference; Unofficial implementation of EAGLE Speculative Decoding☆78Updated 4 months ago
- Utils for Unsloth https://github.com/unslothai/unsloth☆162Updated last week
- Building LLaMA 4 MoE from Scratch☆67Updated 6 months ago
- Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models☆249Updated last year
- ☆51Updated last year
- An innovative library for efficient LLM inference via low-bit quantization☆349Updated last year