CactusQ / TensorRT-LLM-TutorialLinks
Getting started with TensorRT-LLM using BLOOM as a case study
☆23Updated last year
Alternatives and similar repositories for TensorRT-LLM-Tutorial
Users that are interested in TensorRT-LLM-Tutorial are comparing it to the libraries listed below
Sorting:
- Integrating SSE with NVIDIA Triton Inference Server using a Python backend and Zephyr model. There is very less documentation how to use …☆10Updated last year
- Easy and Efficient Quantization for Transformers☆203Updated 5 months ago
- A family of compressed models obtained via pruning and knowledge distillation☆357Updated 3 weeks ago
- ☆317Updated this week
- Notes on quantization in neural networks☆109Updated last year
- Unofficial implementation of https://arxiv.org/pdf/2407.14679☆51Updated last year
- A high-throughput and memory-efficient inference and serving engine for LLMs☆267Updated last year
- This reference can be used with any existing OpenAI integrated apps to run with TRT-LLM inference locally on GeForce GPU on Windows inste…☆127Updated last year
- This repository contains tutorials and examples for Triton Inference Server☆801Updated 2 weeks ago
- A collection of all available inference solutions for the LLMs☆92Updated 8 months ago
- Code for HyperSeg and HyperSum☆16Updated 4 months ago
- A repository dedicated to evaluating the performance of quantizied LLaMA3 using various quantization methods..☆197Updated 10 months ago
- OpenAI compatible API for TensorRT LLM triton backend☆218Updated last year
- This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT)…☆74Updated 2 years ago
- Comparison of Language Model Inference Engines☆236Updated 11 months ago
- A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ/VPTQ, and export to onnx/onnx-runtime easily.☆183Updated 7 months ago
- Lightweight demos for finetuning LLMs. Powered by 🤗 transformers and open-source datasets.☆78Updated last year
- 🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of O…☆320Updated 2 months ago
- For releasing code related to compression methods for transformers, accompanying our publications☆451Updated 10 months ago
- Self-host LLMs with vLLM and BentoML☆157Updated this week
- Distributed training (multi-node) of a Transformer model☆87Updated last year
- The Triton TensorRT-LLM Backend☆910Updated this week
- ☆229Updated 2 months ago
- Enhancing Translation with RAG-Powered Large Language Models☆86Updated last week
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆347Updated 6 months ago
- Complete implementation of Llama2 with/without KV cache & inference 🚀☆48Updated last year
- Parameter-efficient finetuning script for Phi-3-vision, the strong multimodal language model by Microsoft.☆58Updated last year
- Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation☆35Updated last year
- Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models☆257Updated last year
- 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.☆139Updated last year