Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper
โ816Aug 15, 2025Updated last year
Alternatives and similar repositories for native-sparse-attention-pytorch
Users that are interested in native-sparse-attention-pytorch are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ๐ณ Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"โ1,023Feb 5, 2026Updated 8 months ago
- Efficient triton implementation of Native Sparse Attention.โ284May 23, 2025Updated last year
- Implementation of the proposed DeepCrossAttention by Heddes et al at Google research, in Pytorchโ107Apr 3, 2026Updated 6 months ago
- MoBA: Mixture of Block Attention for Long-Context LLMsโ2,194Apr 3, 2025Updated last year
- research impl of Native Sparse Attention (2502.11089)โ63Feb 19, 2025Updated last year
- GPU virtual machines on DigitalOcean Gradient AI โข AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- DeepSeek Native Sparse Attention pytorch implementationโ120Dec 17, 2025Updated 10 months ago
- ๐ Efficient implementations for emerging model architecturesโ5,939Updated this week
- An efficient implementation of the NSA (Native Sparse Attention) kernelโ135Jun 24, 2025Updated last year
- โ37Aug 7, 2025Updated last year
- A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Data Trainingโ954Updated this week
- Explorations into adversarial losses on top of autoregressive loss for language modelingโ41Dec 21, 2025Updated 9 months ago
- Muon is Scalable for LLM Trainingโ1,548Aug 3, 2025Updated last year
- [ICML2025] SpargeAttention: A training-free sparse attention that accelerates any model inference.โ1,263Feb 25, 2026Updated 7 months ago
- Helpful tools and examples for working with flex-attentionโ1,257Updated this week
- Managed Database hosting by DigitalOcean โข AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- DeeperGEMM: crazy optimized versionโ85May 5, 2025Updated last year
- Muon in Int8 Precision Made Possibleโ20Jun 18, 2026Updated 3 months ago
- Unofficial implementation of Titans, SOTA memory for transformers, in Pytorchโ1,992Jul 13, 2026Updated 3 months ago
- Implementation of ๐ Ring Attention, from Liu et al. at Berkeley AI, in Pytorchโ548May 16, 2025Updated last year
- โ32Jul 2, 2025Updated last year
- Design hardware-friendly model architectures and migrate existing LLMs with minimal performance lossโ513Updated this week
- The Gaussian Histogram Loss (HL-Gauss) proposed by Imani et al. with a few convenient wrappers for regression, in Pytorchโ81Sep 2, 2026Updated last month
- โ54May 19, 2025Updated last year
- Mesh generation, skeletonization, voxelization of sparse 3D data.โ26Aug 9, 2026Updated 2 months ago
- Managed Database hosting by DigitalOcean โข AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- a size profiler for cuda binaryโ74Oct 3, 2026Updated 2 weeks ago
- Efficient LLM Inference over Long Sequencesโ393Jun 25, 2025Updated last year
- [ICML 2025] XAttention: Block Sparse Attention with Antidiagonal Scoringโ285Jul 6, 2025Updated last year
- Exploring an idea where one forgets about efficiency and carries out attention across each edge of the nodes (tokens)โ56Mar 25, 2025Updated last year
- FlashMLA: Efficient Multi-head Latent Attention Kernelsโ13,084Updated this week
- Implementation of 2-simplicial attention proposed by Clift et al. (2019) and the recent attempt to make practical in Fast and Simplex, Roโฆโ49Sep 2, 2025Updated last year
- โ65Apr 26, 2025Updated last year
- Fast and memory-efficient exact attentionโ25,151Updated this week
- A bidirectional pipeline parallelism algorithm for computation-communication overlap in DeepSeek V3/R1 training.โ3,018Jan 14, 2026Updated 9 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer โข AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- FLA but cuTileโ27Apr 17, 2026Updated 6 months ago
- Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAIโ300Jun 3, 2025Updated last year
- Quantized Attention on GPUโ45Nov 22, 2024Updated last year
- Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelsโ344Feb 23, 2025Updated last year
- Accelerating MoE with IO and Tile-aware Optimizationsโ780Aug 29, 2026Updated last month
- ไฝฟ็จ cutlass ไปๅบๅจ ada ๆถๆไธๅฎ็ฐ fp8 ็ flash attentionโ83Aug 12, 2024Updated 2 years ago
- A sparse attention kernel supporting mix sparse patternsโ563Jan 18, 2026Updated 9 months ago