Alternatives and similar repositories for selective-attention-transformer

Users that are interested in selective-attention-transformer are comparing it to the libraries listed below

• ablghtianyi / ICL_Modular_Arithmetic
  ☆19Updated 3 months ago
• TianjinYellow / SPAM-Optimizer
  ☆33Updated 4 months ago
• RobertCsordas / moeut
  ☆82Updated 10 months ago
• kuleshov-group / remdm
  Remasking Discrete Diffusion Models with Inference-Time Scaling
  ☆34Updated 4 months ago
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆75Updated 8 months ago
• katiekang1998 / reasoning_generalization
  ☆33Updated 6 months ago
• g-luo / vlm_cross_modal_reps
  Official PyTorch Implementation for Vision-Language Models Create Cross-Modal Task Representations, ICML 2025
  ☆27Updated 2 months ago
• IdoAmos / not-from-scratch
  ☆32Updated 8 months ago
• zhixuan-lin / forgetting-transformer
  [ICLR 2025] Official PyTorch implementation of "Forgetting Transformer: Softmax Attention with a Forget Gate"
  ☆115Updated last week
• formll / resolving-scaling-law-discrepancies
  ☆20Updated last year
• Leiay / looped_transformer
  ☆28Updated last year
• wang-kee / LiNeS
  Official repository of "LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging"
  ☆29Updated 8 months ago
• kyleliang919 / Online-Subspace-Descent
  [NeurIPS 2024] Low rank memory efficient optimizer without SVD
  ☆30Updated 2 weeks ago
• zqOuO / GWT
  ☆13Updated 6 months ago
• chuanyang-Zheng / DAPE
  The this is the official implementation of "DAPE: Data-Adaptive Positional Encoding for Length Extrapolation"
  ☆38Updated 9 months ago
• oripress / EntropyEnigma
  Official code for the ICML 2024 paper "The Entropy Enigma: Success and Failure of Entropy Minimization"
  ☆53Updated last year
• dmis-lab / Monet
  [ICLR 2025] Monet: Mixture of Monosemantic Experts for Transformers
  ☆68Updated 3 weeks ago
• assafbk / DeciMamba
  DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)
  ☆28Updated 3 months ago
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆55Updated 10 months ago
• andyjm3 / SLTrain
  SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)
  ☆32Updated 8 months ago
• goombalab / phi-mamba
  Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…
  ☆110Updated 10 months ago
• krafton-ai / mambaformer-icl
  MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248
  ☆55Updated last year
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆99Updated last week
• Weixin-Liang / Mixture-of-Mamba
  ☆43Updated 5 months ago
• ml-jku / EVA
  One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
  ☆40Updated 9 months ago
• harryjo97 / RDLM
  Official Code Repository for the paper "Continuous Diffusion Model for Language Modeling".
  ☆34Updated 4 months ago
• pixeli99 / MixLN
  [ICLR 2025] Official Pytorch Implementation of "Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN" by Pengxia…
  ☆25Updated 6 months ago
• princeton-pli / what-makes-good-rm
  What Makes a Reward Model a Good Teacher? An Optimization Perspective
  ☆34Updated 3 weeks ago
• VijayLingam95 / SVFT
  ☆30Updated 5 months ago
• BorealisAI / flora-opt
  This is the official repository for the paper "Flora: Low-Rank Adapters Are Secretly Gradient Compressors" in ICML 2024.
  ☆104Updated last year