Alternatives and similar repositories for MomentumSMoE

Users that are interested in MomentumSMoE are comparing it to the libraries listed below

• Ghaith81 / dropkan
  DropKAN (Dropout Kolmogorov Arnold Networks)
  ☆13Updated 10 months ago
• Cohere-Labs-Community / iterative-data-selection
  ☆28Updated 7 months ago
• krafton-ai / mambaformer-icl
  MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248
  ☆54Updated 11 months ago
• Itamarzimm / UnifiedImplicitAttnRepr
  [ICLR 2025] Official Code Release for Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation
  ☆42Updated 3 months ago
• assafbk / DeciMamba
  DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)
  ☆28Updated last month
• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆55Updated 2 months ago
• Weixin-Liang / Mixture-of-Mamba
  ☆43Updated 4 months ago
• Doraemonzzz / hgru2-pytorch
  ☆23Updated 8 months ago
• piotrpiekos / MoSA
  User-friendly implementation of the Mixture-of-Sparse-Attention (MoSA). MoSA selects distinct tokens for each head with expert choice rou…
  ☆17Updated last month
• chuanyang-Zheng / DAPE
  The this is the official implementation of "DAPE: Data-Adaptive Positional Encoding for Length Extrapolation"
  ☆38Updated 7 months ago
• dangxingyu / rnn-icrag
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆27Updated last year
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆54Updated 9 months ago
• gmongaras / Cottention_Transformer
  Code for the paper "Cottention: Linear Transformers With Cosine Attention"
  ☆17Updated 7 months ago
• howard-hou / RWKV-X
  RWKV-X is a Linear Complexity Hybrid Language Model based on the RWKV architecture, integrating Sparse Attention to improve the model's l…
  ☆37Updated last month
• katiekang1998 / reasoning_generalization
  ☆32Updated 5 months ago
• kamanphoebe / Look-into-MoEs
  [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models
  ☆52Updated 4 months ago
• sail-sg / LightTrans
  The official implementation of "LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation"
  ☆20Updated last month
• kuleshov-group / remdm
  Remasking Discrete Diffusion Models with Inference-Time Scaling
  ☆22Updated 3 months ago
• pixeli99 / MixLN
  [ICLR 2025] Official Pytorch Implementation of "Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN" by Pengxia…
  ☆21Updated 5 months ago
• g-luo / vlm_cross_modal_reps
  Official PyTorch Implementation for Vision-Language Models Create Cross-Modal Task Representations, ICML 2025
  ☆26Updated last month
• uclaml / MoE
  Towards Understanding the Mixture-of-Experts Layer in Deep Learning
  ☆30Updated last year
• ml-jku / EVA
  One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
  ☆40Updated 7 months ago
• ablghtianyi / ICL_Modular_Arithmetic
  ☆19Updated 2 months ago
• BlinkDL / LinearAttentionArena
  Here we will test various linear attention designs.
  ☆59Updated last year
• radarFudan / mamba
  ☆17Updated 7 months ago
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆94Updated this week
• TianjinYellow / SPAM-Optimizer
  ☆32Updated 2 months ago
• uclaml / PDE
  Official repo of Progressive Data Expansion: data, code and evaluation
  ☆29Updated last year
• jongwooko / distillm-2
  ☆15Updated 2 months ago
• McGill-NLP / agent-reward-bench
  AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
  ☆15Updated 3 weeks ago