fla-org / native-sparse-attention
🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"
☆601Updated last week
Alternatives and similar repositories for native-sparse-attention:
Users that are interested in native-sparse-attention are comparing it to the libraries listed below
- Muon is Scalable for LLM Training☆993Updated this week
- Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper☆567Updated this week
- Ring attention implementation with flash attention☆717Updated last month
- Muon optimizer: +>30% sample efficiency with <3% wallclock overhead☆529Updated this week
- Efficient LLM Inference over Long Sequences☆365Updated last month
- USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference☆458Updated last week
- TransMLA: Multi-Head Latent Attention Is All You Need☆221Updated 3 weeks ago
- Helpful tools and examples for working with flex-attention☆698Updated last week
- [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads☆442Updated last month
- Understanding R1-Zero-Like Training: A Critical Perspective☆725Updated this week
- 📰 Must-read papers on KV Cache Compression (constantly updating 🤗).☆355Updated this week
- Large Reasoning Models☆800Updated 3 months ago
- ☆396Updated this week
- Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs☆149Updated this week
- Official Repo for Open-Reasoner-Zero☆1,687Updated 3 weeks ago
- LLM KV cache compression made easy☆444Updated last week
- Efficient triton implementation of Native Sparse Attention.☆127Updated this week
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆311Updated 3 months ago
- 🚀 Efficient implementations of state-of-the-art linear attention models in Torch and Triton☆2,177Updated this week
- [NeurIPS'24 Spotlight, ICLR'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which r…☆945Updated last month
- Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels☆858Updated this week
- Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793☆399Updated 3 months ago
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆506Updated 5 months ago
- Puzzles for learning Triton, play it with minimal environment configuration!☆267Updated 3 months ago
- The official implementation of Tensor ProducT ATTenTion Transformer (T6)☆345Updated last month
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models☆439Updated this week
- [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Training☆168Updated last month
- Super-Efficient RLHF Training of LLMs with Parameter Reallocation☆242Updated 2 months ago
- An Open-source RL System from ByteDance Seed and Tsinghua AIR☆915Updated this week
- [ICLR2025 Spotlight🔥] Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters☆541Updated last month