CactusQ / TensorRT-LLM-TutorialLinks
Getting started with TensorRT-LLM using BLOOM as a case study
☆21Updated last year
Alternatives and similar repositories for TensorRT-LLM-Tutorial
Users that are interested in TensorRT-LLM-Tutorial are comparing it to the libraries listed below
Sorting:
- A family of compressed models obtained via pruning and knowledge distillation☆349Updated 9 months ago
- A collection of all available inference solutions for the LLMs☆91Updated 6 months ago
- This reference can be used with any existing OpenAI integrated apps to run with TRT-LLM inference locally on GeForce GPU on Windows inste…☆127Updated last year
- Unofficial implementation of https://arxiv.org/pdf/2407.14679☆48Updated 11 months ago
- Code for HyperSeg and HyperSum☆15Updated last month
- ☆293Updated 3 weeks ago
- A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ/VPTQ, and export to onnx/onnx-runtime easily.☆177Updated 5 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆267Updated 10 months ago
- Efficient LLM Inference over Long Sequences☆390Updated 2 months ago
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆331Updated 4 months ago
- Easy and Efficient Quantization for Transformers☆203Updated 2 months ago
- ☆166Updated 2 weeks ago
- A repository dedicated to evaluating the performance of quantizied LLaMA3 using various quantization methods..☆193Updated 7 months ago
- [ACL 2025 Main] EfficientQAT: Efficient Quantization-Aware Training for Large Language Models☆298Updated 3 months ago
- This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT)…☆70Updated last year
- LLaMA 2 implemented from scratch in PyTorch☆347Updated last year
- Training-free Post-training Efficient Sub-quadratic Complexity Attention. Implemented with OpenAI Triton.☆145Updated this week
- KV cache compression for high-throughput LLM inference☆136Updated 6 months ago
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆310Updated 2 weeks ago
- OpenAI compatible API for TensorRT LLM triton backend☆214Updated last year
- Comparison of Language Model Inference Engines☆229Updated 8 months ago
- Advanced Quantization Algorithm for LLMs and VLMs, with support for CPU, Intel GPU, CUDA and HPU. Seamlessly integrated with Torchao, Tra…☆607Updated last week
- Integrating SSE with NVIDIA Triton Inference Server using a Python backend and Zephyr model. There is very less documentation how to use …☆10Updated last year
- An innovative library for efficient LLM inference via low-bit quantization☆348Updated last year
- Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models☆245Updated last year
- Notes on quantization in neural networks☆97Updated last year
- 3x Faster Inference; Unofficial implementation of EAGLE Speculative Decoding☆75Updated 2 months ago
- For releasing code related to compression methods for transformers, accompanying our publications☆442Updated 7 months ago
- ☆192Updated 8 months ago
- LLM KV cache compression made easy☆596Updated this week