shishishu / LLM-Inference-AccelerationLinks
LLM Inference with Deep Learning Accelerator.
☆44Updated 5 months ago
Alternatives and similar repositories for LLM-Inference-Acceleration
Users that are interested in LLM-Inference-Acceleration are comparing it to the libraries listed below
Sorting:
- Compare different hardware platforms via the Roofline Model for LLM inference tasks.☆100Updated last year
- Summary of some awesome work for optimizing LLM inference☆77Updated 3 weeks ago
- Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of pap…☆255Updated 3 months ago
- Implement some method of LLM KV Cache Sparsity☆32Updated last year
- This repository serves as a comprehensive survey of LLM development, featuring numerous research papers along with their corresponding co…☆154Updated this week
- llm theoretical performance analysis tools and support params, flops, memory and latency analysis.☆96Updated last week
- ☆96Updated 9 months ago
- [ICLR 2025] PEARL: Parallel Speculative Decoding with Adaptive Draft Length☆89Updated 2 months ago
- ☆77Updated 2 months ago
- A GPU-optimized system for efficient long-context LLMs decoding with low-bit KV cache.☆47Updated 2 weeks ago
- Implement Flash Attention using Cute.☆87Updated 6 months ago
- Curated collection of papers in MoE model inference☆197Updated 4 months ago
- ☆141Updated 3 months ago
- ☆62Updated last year
- [USENIX ATC '24] Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hybrid Paral…☆55Updated 10 months ago
- High performance Transformer implementation in C++.☆125Updated 5 months ago
- Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity☆214Updated last year
- ☆77Updated last month
- [ACL 2024] A novel QAT with Self-Distillation framework to enhance ultra low-bit LLMs.☆115Updated last year
- A lightweight design for computation-communication overlap.☆143Updated this week
- ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction (NIPS'24)☆40Updated 6 months ago
- ☆54Updated last year
- Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.☆38Updated 2 weeks ago
- LLM serving cluster simulator☆106Updated last year
- Official implementation of ICML 2024 paper "ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking".☆48Updated 11 months ago
- A standalone GEMM kernel for fp16 activation and quantized weight, extracted from FasterTransformer☆92Updated 3 weeks ago
- [DAC'25] Official implement of "HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference"☆54Updated last week
- LLM Inference analyzer for different hardware platforms☆73Updated 3 weeks ago
- ☆36Updated 10 months ago
- An easy-to-use package for implementing SmoothQuant for LLMs☆100Updated 2 months ago