zhihu / TLLM_QMMLinks
TLLM_QMM strips the implementation of quantized kernels of Nvidia's TensorRT-LLM, removing NVInfer dependency and exposes ease of use Pytorch module. We modified the dequantation and weight preprocessing to align with popular quantization alogirthms such as AWQ and GPTQ, and combine them with new FP8 quantization.
☆16Updated last year
Alternatives and similar repositories for TLLM_QMM
Users that are interested in TLLM_QMM are comparing it to the libraries listed below
Sorting:
- TePDist (TEnsor Program DISTributed) is an HLO-level automatic distributed system for DL models.☆97Updated 2 years ago
- PyTorch distributed training acceleration framework☆53Updated 3 months ago
- ☆130Updated 10 months ago
- ☆511Updated 2 months ago
- Compare different hardware platforms via the Roofline Model for LLM inference tasks.☆119Updated last year
- A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.☆114Updated 5 months ago
- Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.☆270Updated 2 years ago
- GLake: optimizing GPU memory management and IO transmission.☆489Updated 7 months ago
- DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …☆267Updated 3 months ago
- DeepRec Extension is an easy-to-use, stable and efficient large-scale distributed training system based on DeepRec.☆11Updated last year
- RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.☆920Updated this week
- AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and ver…☆267Updated 2 months ago
- ☆152Updated 10 months ago
- A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster☆159Updated last year
- ☆58Updated 5 years ago
- DeepXTrace is a lightweight tool for precisely diagnosing slow ranks in DeepEP-based environments.☆66Updated last week
- KV cache store for distributed LLM inference☆358Updated 2 months ago
- Fast and memory-efficient exact attention☆97Updated last week
- ☆46Updated 11 months ago
- ☆139Updated last year
- HierarchicalKV is a part of NVIDIA Merlin and provides hierarchical key-value storage to meet RecSys requirements. The key capability of…☆175Updated last week
- ☆219Updated 2 years ago
- A high-performance serving system for DeepRec based on TensorFlow Serving.☆19Updated 2 years ago
- ☆75Updated 11 months ago
- ☆97Updated 7 months ago
- [USENIX ATC '24] Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hybrid Paral…☆66Updated last year
- ☆149Updated 8 months ago
- The driver for LMCache core to run in vLLM☆56Updated 9 months ago
- ☆125Updated last week
- ☆44Updated 8 months ago