NonvolatileMemory / flash_tree_attnLinks
☆18Updated 8 months ago
Alternatives and similar repositories for flash_tree_attn
Users that are interested in flash_tree_attn are comparing it to the libraries listed below
Sorting:
- SQUEEZED ATTENTION: Accelerating Long Prompt LLM Inference☆51Updated 9 months ago
- Quantized Attention on GPU☆44Updated 9 months ago
- ☆50Updated 3 months ago
- LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification☆62Updated last month
- Pytorch implementation of our paper accepted by ICML 2024 -- CaM: Cache Merging for Memory-efficient LLMs Inference☆42Updated last year
- 16-fold memory access reduction with nearly no loss☆104Updated 5 months ago
- [COLM 2024] SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models☆24Updated 10 months ago
- Beyond KV Caching: Shared Attention for Efficient LLMs☆19Updated last year
- [ICLR 2025] TidalDecode: A Fast and Accurate LLM Decoding with Position Persistent Sparse Attention☆47Updated 2 weeks ago
- Accelerate LLM preference tuning via prefix sharing with a single line of code☆43Updated last month
- Multi-Candidate Speculative Decoding☆36Updated last year
- PoC for "SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning" [arXiv '25]☆48Updated last month
- Code for paper: [ICLR2025 Oral] FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference☆133Updated 3 months ago
- ☆123Updated 2 months ago
- Estimate MFU for DeepSeekV3☆24Updated 7 months ago
- Source code for the paper "LongGenBench: Long-context Generation Benchmark"☆23Updated 10 months ago
- [ICLR 2025] DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference☆32Updated 2 months ago
- An efficient implementation of the NSA (Native Sparse Attention) kernel☆113Updated 2 months ago
- ☆21Updated 5 months ago
- LLM Inference with Microscaling Format☆27Updated 9 months ago
- Official Implementation of SAM-Decoding: Speculative Decoding via Suffix Automaton☆31Updated 6 months ago
- official code for GliDe with a CaPE☆18Updated last year
- Efficient Attention Methods: A Comprehensive Survey☆75Updated this week
- ☆81Updated 7 months ago
- The Official Implementation of Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference☆89Updated 2 months ago
- ☆61Updated 4 months ago
- ☆56Updated 8 months ago
- QAQ: Quality Adaptive Quantization for LLM KV Cache☆52Updated last year
- [ACL 2024] Official PyTorch implementation of "IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact"☆47Updated last year
- [ICML 2025] SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity☆51Updated last month