ModelTC / awesome-lm-systemLinks
Summary of system papers/frameworks/codes/tools on training or serving large model
☆57Updated last year
Alternatives and similar repositories for awesome-lm-system
Users that are interested in awesome-lm-system are comparing it to the libraries listed below
Sorting:
- ☆77Updated 2 months ago
- Odysseus: Playground of LLM Sequence Parallelism☆70Updated last year
- An easy-to-use package for implementing SmoothQuant for LLMs☆102Updated 3 months ago
- ☆96Updated 10 months ago
- Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.☆38Updated last month
- ☆87Updated 3 months ago
- ☆74Updated last month
- GPU operators for sparse tensor operations☆33Updated last year
- ☆49Updated last month
- ☆77Updated 5 months ago
- Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity☆216Updated last year
- [NeurIPS 2024] The official implementation of "Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exitin…☆58Updated last year
- Performance of the C++ interface of flash attention and flash attention v2 in large language model (LLM) inference scenarios.☆39Updated 4 months ago
- PyTorch bindings for CUTLASS grouped GEMM.☆101Updated last month
- A simple calculation for LLM MFU.☆39Updated 4 months ago
- Quantized Attention on GPU☆44Updated 7 months ago
- [ACL 2024] A novel QAT with Self-Distillation framework to enhance ultra low-bit LLMs.☆115Updated last year
- QQQ is an innovative and hardware-optimized W4A8 quantization solution for LLMs.☆133Updated 3 months ago
- GPTQ inference TVM kernel☆40Updated last year
- A standalone GEMM kernel for fp16 activation and quantized weight, extracted from FasterTransformer☆92Updated last week
- ☆86Updated 3 years ago
- llama INT4 cuda inference with AWQ☆54Updated 5 months ago
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆165Updated last year
- Boosting 4-bit inference kernels with 2:4 Sparsity☆80Updated 10 months ago
- ☆60Updated 2 months ago
- Framework to reduce autotune overhead to zero for well known deployments.☆79Updated last week
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆313Updated last year
- 16-fold memory access reduction with nearly no loss☆100Updated 3 months ago
- QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference☆117Updated last year
- [ICLR 2025] PEARL: Parallel Speculative Decoding with Adaptive Draft Length☆93Updated 3 months ago