microsoft / chunk-attention
☆41Updated 5 months ago
Related projects ⓘ
Alternatives and complementary repositories for chunk-attention
- Summary of system papers/frameworks/codes/tools on training or serving large model☆56Updated 10 months ago
- Odysseus: Playground of LLM Sequence Parallelism☆53Updated 4 months ago
- ☆63Updated 3 months ago
- ☆70Updated 2 years ago
- Quantized Attention on GPU☆29Updated this week
- ☆79Updated 2 months ago
- Official PyTorch implementation of FlatQuant: Flatness Matters for LLM Quantization☆57Updated this week
- ☆36Updated last month
- PyTorch implementation of paper "Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline".☆74Updated last year
- PyTorch bindings for CUTLASS grouped GEMM.☆51Updated last week
- Sequence-level 1F1B schedule for LLMs.☆17Updated 5 months ago
- ☆42Updated 7 months ago
- Materials for learning SGLang☆75Updated this week
- [ACL 2024] A novel QAT with Self-Distillation framework to enhance ultra low-bit LLMs.☆79Updated 5 months ago
- Official implementation of ICML 2024 paper "ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking".☆40Updated 3 months ago
- PyTorch bindings for CUTLASS grouped GEMM.☆67Updated 3 months ago
- ATC23 AE☆43Updated last year
- Performance of the C++ interface of flash attention and flash attention v2 in large language model (LLM) inference scenarios.☆26Updated 2 months ago
- Boosting 4-bit inference kernels with 2:4 Sparsity☆51Updated 2 months ago
- Decoding Attention is specially optimized for multi head attention (MHA) using CUDA core for the decoding stage of LLM inference.☆24Updated this week
- The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.☆35Updated 3 weeks ago
- QQQ is an innovative and hardware-optimized W4A8 quantization solution for LLMs.☆75Updated 3 weeks ago
- Multi-Candidate Speculative Decoding☆28Updated 6 months ago
- The official code for paper "parallel speculative decoding with adaptive draft length."☆21Updated 2 months ago
- ☆55Updated 5 months ago
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)☆74Updated 3 weeks ago
- Official Repo for SparseLLM: Global Pruning of LLMs (NeurIPS 2024)☆34Updated 2 weeks ago
- QAQ: Quality Adaptive Quantization for LLM KV Cache☆42Updated 7 months ago
- ☆42Updated last month
- An easy-to-use package for implementing SmoothQuant for LLMs☆82Updated 5 months ago