fla-org / native-sparse-attention
🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"
☆636Updated last month
Alternatives and similar repositories for native-sparse-attention:
Users that are interested in native-sparse-attention are comparing it to the libraries listed below
- Ring attention implementation with flash attention☆743Updated 2 weeks ago
- Muon is Scalable for LLM Training☆1,029Updated 3 weeks ago
- Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper☆601Updated 3 weeks ago
- USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference☆477Updated this week
- [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads☆453Updated 2 months ago
- Helpful tools and examples for working with flex-attention☆726Updated last week
- Efficient LLM Inference over Long Sequences☆368Updated this week
- Understanding R1-Zero-Like Training: A Critical Perspective☆882Updated last week
- Muon optimizer: +>30% sample efficiency with <3% wallclock overhead☆577Updated last month
- Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs☆161Updated this week
- TransMLA: Multi-Head Latent Attention Is All You Need☆238Updated last month
- ☆179Updated last week
- VeOmni: Scaling any Modality Model Training to any Accelerators with PyTorch native Training Framework☆297Updated 2 weeks ago
- LLM KV cache compression made easy☆458Updated last week
- Efficient triton implementation of Native Sparse Attention.☆139Updated 2 weeks ago
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆511Updated 5 months ago
- [ICML 2024] CLLMs: Consistency Large Language Models☆391Updated 5 months ago
- 📰 Must-read papers on KV Cache Compression (constantly updating 🤗).☆376Updated 2 weeks ago
- ☆698Updated this week
- ☆419Updated this week
- Super-Efficient RLHF Training of LLMs with Parameter Reallocation☆281Updated 3 months ago
- Distributed Triton for Parallel Systems☆451Updated 2 weeks ago
- A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Data Training☆248Updated this week
- A sparse attention kernel supporting mix sparse patterns☆197Updated 2 months ago
- [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Training☆184Updated last week
- Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models☆279Updated 2 months ago
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆715Updated 6 months ago
- Large Context Attention☆704Updated 3 months ago
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆317Updated 4 months ago
- DeepSeek Native Sparse Attention pytorch implementation☆62Updated last month