Alternatives and similar repositories for Active-Dormant-Attention

Users that are interested in Active-Dormant-Attention are comparing it to the libraries listed below

• fangyuan-ksgk / selective-attention-transformer
  Unofficial Implementation of Selective Attention Transformer
  ☆17Updated last year
• andyjm3 / SLTrain
  SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)
  ☆36Updated last year
• sjelassi / transformers_ssm_copy
  ☆35Updated last year
• ablghtianyi / ICL_Modular_Arithmetic
  ☆19Updated 7 months ago
• sail-sg / Attention-Sink
  [ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)
  ☆135Updated 4 months ago
• qiuzh20 / gated_attention
  The official implementation for [NeurIPS2025 Oral] Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink…
  ☆105Updated 2 months ago
• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆61Updated 4 months ago
• alvin-zyl / CoLA
  Implementation of CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
  ☆24Updated 9 months ago
• formll / resolving-scaling-law-discrepancies
  ☆20Updated 2 weeks ago
• Infini-AI-Lab / Kinetics
  Kinetics: Rethinking Test-Time Scaling Laws
  ☆82Updated 4 months ago
• nasosger / MuToR
  [NeurIPS '25] Multi-Token Prediction Needs Registers
  ☆24Updated 2 months ago
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆85Updated last year
• deep-spin / adasplash
  AdaSplash: Adaptive Sparse Flash Attention (aka Flash Entmax Attention)
  ☆29Updated last month
• osehmathias / lisa
  LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
  ☆35Updated last year
• Leiay / looped_transformer
  ☆33Updated last year
• PKU-ML / LongPPL
  Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"
  ☆105Updated last month
• zyushun / hessian-spectrum
  Code for the paper: Why Transformers Need Adam: A Hessian Perspective
  ☆63Updated 8 months ago
• hahnyuan / ASVD4LLM
  Activation-aware Singular Value Decomposition for Compressing Large Language Models
  ☆80Updated last year
• alessiodevoto / l2compress
  Code for the EMNLP24 paper "A simple and effective L2 norm based method for KV Cache compression."
  ☆17Updated 11 months ago
• socialfoundations / tttlm
  Test-time-training on nearest neighbors for large language models
  ☆46Updated last year
• Lucky-Lance / SPP
  [ICML 2024] SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
  ☆21Updated last year
• pilancilab / Riemannian_Preconditioned_LoRA
  source code for paper "Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models"
  ☆32Updated last year
• HanseulJo / position-coupling
  Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure (NeurIPS 2024) + Arithmetic Transfor…
  ☆11Updated 3 weeks ago
• tim-lawson / mlsae
  Multi-Layer Sparse Autoencoders (ICLR 2025)
  ☆26Updated 9 months ago
• LIONS-EPFL / scion
  ☆47Updated 3 weeks ago
• holarissun / RewardModelingBeyondBradleyTerry
  official implementation of ICLR'2025 paper: Rethinking Bradley-Terry Models in Preference-based Reward Modeling: Foundations, Theory, and…
  ☆69Updated 7 months ago
• bethgelab / sober-reasoning
  A Sober Look at Language Model Reasoning
  ☆87Updated this week
• imagination-research / LCSC
  [ICLR 2025] Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better
  ☆16Updated 9 months ago
• OpenSparseLLMs / Linearization
  ☆61Updated 4 months ago
• sustcsonglin / linear-attention-and-beyond-slides
  ☆95Updated 8 months ago