zhihu / TLLM_QMMLinks
TLLM_QMM strips the implementation of quantized kernels of Nvidia's TensorRT-LLM, removing NVInfer dependency and exposes ease of use Pytorch module. We modified the dequantation and weight preprocessing to align with popular quantization alogirthms such as AWQ and GPTQ, and combine them with new FP8 quantization.
☆16Updated last year
Alternatives and similar repositories for TLLM_QMM
Users that are interested in TLLM_QMM are comparing it to the libraries listed below
Sorting:
- PyTorch distributed training acceleration framework☆53Updated last month
- Compare different hardware platforms via the Roofline Model for LLM inference tasks.☆115Updated last year
- ☆504Updated 3 weeks ago
- TePDist (TEnsor Program DISTributed) is an HLO-level automatic distributed system for DL models.☆96Updated 2 years ago
- A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.☆111Updated 4 months ago
- ☆129Updated 9 months ago
- Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.☆268Updated 2 years ago
- ☆46Updated 9 months ago
- ☆150Updated 8 months ago
- GLake: optimizing GPU memory management and IO transmission.☆480Updated 6 months ago
- AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and ver…☆266Updated last month
- DeepRec Extension is an easy-to-use, stable and efficient large-scale distributed training system based on DeepRec.☆11Updated last year
- Fast and memory-efficient exact attention☆95Updated this week
- DeepXTrace is a lightweight tool for precisely diagnosing slow ranks in DeepEP-based environments.☆54Updated last week
- KV cache store for distributed LLM inference☆336Updated 3 weeks ago
- ☆141Updated last year
- ☆95Updated 6 months ago
- ☆199Updated 4 months ago
- Pipeline Parallelism Emulation and Visualization☆67Updated 3 months ago
- A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster☆158Updated last year
- DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …☆265Updated last month
- HierarchicalKV is a part of NVIDIA Merlin and provides hierarchical key-value storage to meet RecSys requirements. The key capability of…☆171Updated last week
- A lightweight design for computation-communication overlap.☆177Updated 2 weeks ago
- DeepSeek-V3/R1 inference performance simulator☆170Updated 6 months ago
- Performance of the C++ interface of flash attention and flash attention v2 in large language model (LLM) inference scenarios.☆40Updated 7 months ago
- ☆88Updated last week
- [USENIX ATC '24] Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hybrid Paral…☆65Updated last year
- ☆220Updated 2 years ago
- ☆298Updated last week
- A high-performance serving system for DeepRec based on TensorFlow Serving.☆19Updated last year