SqueezeBits / Torch-TRTLLMLinks
Ditto is an open-source framework that enables direct conversion of HuggingFace PreTrainedModels into TensorRT-LLM engines.
☆49Updated 2 months ago
Alternatives and similar repositories for Torch-TRTLLM
Users that are interested in Torch-TRTLLM are comparing it to the libraries listed below
Sorting:
- QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference☆118Updated last year
- Easy and Efficient Quantization for Transformers☆203Updated 2 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆266Updated 11 months ago
- Boosting 4-bit inference kernels with 2:4 Sparsity☆82Updated last year
- ☆27Updated last year
- ☆199Updated 4 months ago
- Inference code for LLaMA models☆21Updated 5 months ago
- ☆296Updated last week
- ☆74Updated 5 months ago
- Fast low-bit matmul kernels in Triton☆371Updated last week
- The official implementation of the EMNLP 2023 paper LLM-FP4☆214Updated last year
- A safetensors extension to efficiently store sparse quantized tensors on disk☆161Updated this week
- Training-free Post-training Efficient Sub-quadratic Complexity Attention. Implemented with OpenAI Triton.☆147Updated last week
- ArcticTraining is a framework designed to simplify and accelerate the post-training process for large language models (LLMs)☆218Updated this week
- Efficient LLM Inference over Long Sequences☆391Updated 2 months ago
- a minimal cache manager for PagedAttention, on top of llama3.☆120Updated last year
- ☆121Updated last year
- Summary of system papers/frameworks/codes/tools on training or serving large model☆57Updated last year
- A performance library for machine learning applications.☆184Updated last year
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆127Updated 9 months ago
- [ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models☆23Updated last year
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆42Updated last year
- QQQ is an innovative and hardware-optimized W4A8 quantization solution for LLMs.☆139Updated last month
- ☆217Updated 8 months ago
- vLLM adapter for a TGIS-compatible gRPC server.☆40Updated this week
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆320Updated last year
- Framework to reduce autotune overhead to zero for well known deployments.☆82Updated this week
- Examples for MS-AMP package.☆30Updated 2 months ago
- A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ/VPTQ, and export to onnx/onnx-runtime easily.☆177Updated 5 months ago
- Official PyTorch implementation of "GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance" (ICML 2025)☆44Updated 2 months ago