zhihu / TLLM_QMMLinks
TLLM_QMM strips the implementation of quantized kernels of Nvidia's TensorRT-LLM, removing NVInfer dependency and exposes ease of use Pytorch module. We modified the dequantation and weight preprocessing to align with popular quantization alogirthms such as AWQ and GPTQ, and combine them with new FP8 quantization.
☆16Updated last year
Alternatives and similar repositories for TLLM_QMM
Users that are interested in TLLM_QMM are comparing it to the libraries listed below
Sorting:
- TePDist (TEnsor Program DISTributed) is an HLO-level automatic distributed system for DL models.☆95Updated 2 years ago
- PyTorch distributed training acceleration framework☆52Updated last month
- Compare different hardware platforms via the Roofline Model for LLM inference tasks.☆112Updated last year
- A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.☆108Updated 3 months ago
- ☆128Updated 8 months ago
- AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and ver…☆263Updated 3 weeks ago
- ☆491Updated last month
- HierarchicalKV is a part of NVIDIA Merlin and provides hierarchical key-value storage to meet RecSys requirements. The key capability of…☆169Updated last week
- ☆46Updated 9 months ago
- GLake: optimizing GPU memory management and IO transmission.☆479Updated 5 months ago
- ☆150Updated 8 months ago
- KV cache store for distributed LLM inference☆326Updated 3 months ago
- DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …☆264Updated last month
- High performance RDMA-based distributed feature collection component for training GNN model on EXTREMELY large graph☆55Updated 3 years ago
- A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster☆157Updated last year
- DeepXTrace is a lightweight tool for precisely diagnosing slow ranks in DeepEP-based environments.☆47Updated last week
- Fast and memory-efficient exact attention☆93Updated this week
- The driver for LMCache core to run in vLLM☆49Updated 7 months ago
- Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.☆268Updated 2 years ago
- ☆94Updated 5 months ago
- DeepSeek-V3/R1 inference performance simulator☆165Updated 5 months ago
- ☆58Updated 5 years ago
- Pipeline Parallelism Emulation and Visualization☆65Updated 3 months ago
- optimized BERT transformer inference on NVIDIA GPU. https://arxiv.org/abs/2210.03052☆476Updated last year
- NCCL Fast Socket is a transport layer plugin to improve NCCL collective communication performance on Google Cloud.☆121Updated last year
- DeepRec Extension is an easy-to-use, stable and efficient large-scale distributed training system based on DeepRec.☆11Updated last year
- A library developed by Volcano Engine for high-performance reading and writing of PyTorch model files.☆22Updated 8 months ago
- RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.☆847Updated last month
- [USENIX ATC '24] Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hybrid Paral…☆61Updated last year
- ☆98Updated last year