Alternatives and similar repositories for SparseAttention:

Users that are interested in SparseAttention are comparing it to the libraries listed below

• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆51Updated 2 months ago
• ambisinister / mla-experiments
  Experiments on Multi-Head Latent Attention
  ☆76Updated 7 months ago
• kyegomez / FlashAttention20
  Get down and dirty with FlashAttention2.0 in pytorch, plug in and play no complex CUDA kernels
  ☆102Updated last year
• insuhan / hyper-attn
  ☆81Updated last year
• kyegomez / SwitchTransformers
  Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficien…
  ☆89Updated 2 months ago
• kyegomez / MoE-Mamba
  Implementation of MoE Mamba from the paper: "MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts" in Pytorch and Ze…
  ☆101Updated 2 months ago
• pengzhangzhi / Awesome-Mamba
  Awesome list of papers that extend Mamba to various applications.
  ☆132Updated 3 months ago
• xiayuqing0622 / flex_head_fa
  Fast and memory-efficient exact attention
  ☆67Updated 3 weeks ago
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆117Updated 5 months ago
• goombalab / phi-mamba
  Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…
  ☆102Updated 6 months ago
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆86Updated 2 weeks ago
• kyegomez / Blockwise-Parallel-Transformer
  32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.
  ☆46Updated last year
• goombalab / hydra
  Official implementation of "Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers"
  ☆125Updated last month
• lucidrains / hyper-connections
  Attempt to make multiple residual streams from Bytedance's Hyper-Connections paper accessible to the public
  ☆78Updated last month
• tanaymeh / mamba-train
  A single repo with all scripts and utils to train / fine-tune the Mamba model with or without FIM
  ☆54Updated 11 months ago
• lucidrains / agent-attention-pytorch
  Implementation of Agent Attention in Pytorch
  ☆90Updated 8 months ago
• krafton-ai / mambaformer-icl
  MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248
  ☆51Updated 9 months ago
• thu-ml / ReMoE
  [ICLR2025] Codebase for "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing", built on Megatron-LM.
  ☆65Updated 3 months ago
• kyegomez / Jamba
  PyTorch Implementation of Jamba: "Jamba: A Hybrid Transformer-Mamba Language Model"
  ☆160Updated 2 months ago
• test-time-training / ttt-lm-kernels
  Inference Speed Benchmark for Learning to (Learn at Test Time): RNNs with Expressive Hidden States
  ☆64Updated 8 months ago
• lucidrains / infini-transformer-pytorch
  Implementation of Infini-Transformer in Pytorch
  ☆110Updated 2 months ago
• fla-org / flame
  🔥 A minimal training framework for scaling FLA models
  ☆82Updated this week
• alxndrTL / othello_mamba
  Evaluating the Mamba architecture on the Othello game
  ☆46Updated 11 months ago
• hahnyuan / ASVD4LLM
  Activation-aware Singular Value Decomposition for Compressing Large Language Models
  ☆59Updated 5 months ago
• epfml / dynamic-sparse-flash-attention
  ☆141Updated last year
• OpenNLPLab / lightning-attention
  Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  ☆272Updated last month
• NX-AI / mlstm_kernels
  Tiled Flash Linear Attention library for fast and efficient mLSTM Kernels.
  ☆47Updated last week
• lucidrains / PEER-pytorch
  Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind
  ☆121Updated 7 months ago
• WailordHe / DenseSSM
  A repository for DenseSSMs
  ☆87Updated 11 months ago
• HazyResearch / flash-fft-conv
  FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
  ☆307Updated 3 months ago