Alternatives and similar repositories for native-sparse-attention

Users that are interested in native-sparse-attention are comparing it to the libraries listed below

• zhuzilin / ring-flash-attention
  Ring attention implementation with flash attention
  ☆949Updated 3 months ago
• lucidrains / native-sparse-attention-pytorch
  Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper
  ☆790Updated 4 months ago
• MoonshotAI / Moonlight
  Muon is Scalable for LLM Training
  ☆1,387Updated 4 months ago
• feifeibear / long-context-attention
  USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
  ☆618Updated this week
• meta-pytorch / attention-gym
  Helpful tools and examples for working with flex-attention
  ☆1,095Updated this week
• NVlabs / GatedDeltaNet
  [ICLR 2025] Official PyTorch Implementation of Gated Delta Networks: Improving Mamba2 with Delta Rule
  ☆404Updated 3 months ago
• ByteDance-Seed / VeOmni
  VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo
  ☆1,451Updated this week
• NVlabs / Fast-dLLM
  Official implementation of "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding"
  ☆748Updated 3 weeks ago
• MuLabPKU / TransMLA
  TransMLA: Multi-Head Latent Attention Is All You Need (NeurIPS 2025 Spotlight)
  ☆419Updated 3 months ago
• XunhaoLai / native-sparse-attention-triton
  Efficient triton implementation of Native Sparse Attention.
  ☆256Updated 7 months ago
• fla-org / flame
  🔥 A minimal training framework for scaling FLA models
  ☆324Updated last month
• QwenLM / ParScale
  Parallel Scaling Law for Language Model — Beyond Parameter and Inference Time Scaling
  ☆465Updated 7 months ago
• SandAI-org / MagiAttention
  A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Data Training
  ☆590Updated last week
• mit-han-lab / duo-attention
  [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
  ☆515Updated 10 months ago
• stepfun-ai / Step3
  ☆440Updated 4 months ago
• NVIDIA / Star-Attention
  Efficient LLM Inference over Long Sequences
  ☆394Updated 6 months ago
• JT-Ushio / MHA2MLA
  Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
  ☆198Updated 3 weeks ago
• NVIDIA-NeMo / RL
  Scalable toolkit for efficient model reinforcement
  ☆1,171Updated this week
• tensorgi / TPA
  [NeurIPS 2025 Spotlight] TPA: Tensor ProducT ATTenTion Transformer (T6) (https://arxiv.org/abs/2501.06425)
  ☆438Updated last week
• NVIDIA / kvpress
  LLM KV cache compression made easy
  ☆729Updated last week
• haoliuhl / ringattention
  Large Context Attention
  ☆757Updated 2 months ago
• MoonshotAI / MoBA
  MoBA: Mixture of Block Attention for Long-Context LLMs
  ☆2,022Updated 8 months ago
• apple / ml-cross-entropy
  ☆568Updated 3 months ago
• Dao-AILab / sonic-moe
  Accelerating MoE with IO and Tile-aware Optimizations
  ☆469Updated this week
• lucidrains / ring-attention-pytorch
  Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
  ☆549Updated 7 months ago
• sgl-project / SpecForge
  Train speculative decoding models effortlessly and port them smoothly to SGLang serving.
  ☆577Updated this week
• radixark / miles
  ☆629Updated this week
• OpenNLPLab / lightning-attention
  Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  ☆338Updated 10 months ago
• THUDM / slime
  slime is an LLM post-training framework for RL Scaling.
  ☆3,022Updated this week
• thinking-machines-lab / batch_invariant_ops
  ☆941Updated last month