microsoft / chunk-attention
☆43Updated 6 months ago
Related projects ⓘ
Alternatives and complementary repositories for chunk-attention
- Summary of system papers/frameworks/codes/tools on training or serving large model☆56Updated 11 months ago
- Odysseus: Playground of LLM Sequence Parallelism☆57Updated 5 months ago
- ☆38Updated last month
- Official implementation of ICML 2024 paper "ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking".☆41Updated 4 months ago
- ☆64Updated 4 months ago
- ☆42Updated 7 months ago
- PyTorch implementation of paper "Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline".☆74Updated last year
- ☆70Updated 2 years ago
- PyTorch bindings for CUTLASS grouped GEMM.☆53Updated 3 weeks ago
- [ACL 2024] A novel QAT with Self-Distillation framework to enhance ultra low-bit LLMs.☆85Updated 6 months ago
- [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts☆34Updated 8 months ago
- PyTorch bindings for CUTLASS grouped GEMM.☆68Updated 4 months ago
- GPTQ inference TVM kernel☆36Updated 6 months ago
- The official code for paper "parallel speculative decoding with adaptive draft length."☆25Updated 3 months ago
- Boosting 4-bit inference kernels with 2:4 Sparsity☆51Updated 2 months ago
- Implementation of Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting☆44Updated 4 months ago
- Quantized Attention on GPU☆31Updated this week
- Official PyTorch implementation of FlatQuant: Flatness Matters for LLM Quantization☆66Updated last week
- Multi-Candidate Speculative Decoding☆28Updated 7 months ago
- Sequence-level 1F1B schedule for LLMs.☆17Updated 5 months ago
- ☆79Updated 2 months ago
- Materials for learning SGLang☆110Updated this week
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆203Updated this week
- The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.☆39Updated last month
- QAQ: Quality Adaptive Quantization for LLM KV Cache☆42Updated 7 months ago
- An easy-to-use package for implementing SmoothQuant for LLMs☆83Updated 6 months ago
- Decoding Attention is specially optimized for multi head attention (MHA) using CUDA core for the decoding stage of LLM inference.☆26Updated 2 weeks ago
- Pytorch implementation of our paper accepted by ICML 2024 -- CaM: Cache Merging for Memory-efficient LLMs Inference☆28Updated 5 months ago
- ☆189Updated 6 months ago
- MagicPIG: LSH Sampling for Efficient LLM Generation☆59Updated 3 weeks ago