Alternatives and similar repositories for TensorRT-LLM-Tutorial:

Users that are interested in TensorRT-LLM-Tutorial are comparing it to the libraries listed below

• mani-kantap / llm-inference-solutions
  A collection of all available inference solutions for the LLMs
  ☆74Updated 4 months ago
• triton-inference-server / vllm_backend
  ☆216Updated last week
• wejoncy / QLLM
  A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ, and export to onnx/onnx-runtime easily.
  ☆153Updated this week
• alperiox / Compact-Language-Models-via-Pruning-and-Knowledge-Distillation
  Unofficial implementation of https://arxiv.org/pdf/2407.14679
  ☆41Updated 4 months ago
• unslothai / unsloth-zoo
  Utils for Unsloth
  ☆27Updated this week
• Macaronlin / LLaMA3-Quantization
  A repository dedicated to evaluating the performance of quantizied LLaMA3 using various quantization methods..
  ☆175Updated last week
• NVIDIA / kvpress
  LLM KV cache compression made easy
  ☆305Updated last week
• NetEase-FuXi / EETQ
  Easy and Efficient Quantization for Transformers
  ☆191Updated last month
• TRT2022 / trtllm-llama
  ☢️ TensorRT 2023复赛——基于TensorRT-LLM的Llama模型推断加速优化
  ☆45Updated last year
• NVIDIA / Star-Attention
  Efficient LLM Inference over Long Sequences
  ☆347Updated 3 weeks ago
• NVIDIA / trt-llm-as-openai-windows
  This reference can be used with any existing OpenAI integrated apps to run with TRT-LLM inference locally on GeForce GPU on Windows inste…
  ☆117Updated 10 months ago
• NVlabs / Minitron
  A family of compressed models obtained via pruning and knowledge distillation
  ☆309Updated 2 months ago
• premAI-io / benchmarks
  🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
  ☆138Updated 5 months ago
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆186Updated 5 months ago
• neuralmagic / AutoFP8
  ☆168Updated 3 months ago
• hkproj / pytorch-transformer-distributed
  Distributed training (multi-node) of a Transformer model
  ☆49Updated 9 months ago
• LLM-inference-router / vllm-router
  vLLM Router
  ☆17Updated 10 months ago
• pphuc25 / distil-cd
  Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation
  ☆33Updated 10 months ago
• nyunAI / PruneGPT
  ☆52Updated 7 months ago
• neuralmagic / guidellm
  Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
  ☆183Updated last month
• InternLM / turbomind
  ☆55Updated last month
• triton-inference-server / tensorrt_backend
  The Triton backend for TensorRT.
  ☆68Updated last week
• neuralmagic / nm-vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆257Updated 3 months ago
• facebookresearch / LayerSkip
  Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024
  ☆259Updated last week
• GaiZhenbiao / Phi3V-Finetuning
  Parameter-efficient finetuning script for Phi-3-vision, the strong multimodal language model by Microsoft.
  ☆56Updated 7 months ago
• tspeterkim / mixed-precision-from-scratch
  Mixed precision training from scratch with Tensors and CUDA
  ☆21Updated 8 months ago
• Infini-AI-Lab / MagicDec
  Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆107Updated last month
• hkproj / triton-flash-attention
  ☆95Updated 2 weeks ago
• jshuadvd / LongRoPE
  Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
  ☆125Updated 6 months ago
• nexusflowai / NexusBench
  Nexusflow function call, tool use, and agent benchmarks.
  ☆18Updated last month