Alternatives and similar repositories for Active-Dormant-Attention

Users that are interested in Active-Dormant-Attention are comparing it to the libraries listed below

• fangyuan-ksgk / selective-attention-transformer
  Unofficial Implementation of Selective Attention Transformer
  ☆20Updated last year
• sjelassi / transformers_ssm_copy
  ☆35Updated last year
• andyjm3 / SLTrain
  SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)
  ☆39Updated last year
• ablghtianyi / ICL_Modular_Arithmetic
  ☆19Updated 9 months ago
• sail-sg / Attention-Sink
  [ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)
  ☆152Updated 6 months ago
• zyushun / hessian-spectrum
  Code for the paper: Why Transformers Need Adam: A Hessian Perspective
  ☆63Updated 10 months ago
• nasosger / MuToR
  [NeurIPS '25] Multi-Token Prediction Needs Registers
  ☆26Updated last month
• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆62Updated 6 months ago
• formll / resolving-scaling-law-discrepancies
  ☆20Updated 2 months ago
• Infini-AI-Lab / Kinetics
  Kinetics: Rethinking Test-Time Scaling Laws
  ☆85Updated 6 months ago
• PKU-ML / LongPPL
  Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"
  ☆109Updated 3 months ago
• LIONS-EPFL / scion
  ☆52Updated last month
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆87Updated last year
• alvin-zyl / CoLA
  Implementation of CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
  ☆25Updated 11 months ago
• sustcsonglin / linear-attention-and-beyond-slides
  ☆104Updated 10 months ago
• Lucky-Lance / SPP
  [ICML 2024] SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
  ☆21Updated last year
• alessiodevoto / l2compress
  Code for the EMNLP24 paper "A simple and effective L2 norm based method for KV Cache compression."
  ☆18Updated last year
• bethgelab / sober-reasoning
  A Sober Look at Language Model Reasoning
  ☆92Updated 2 months ago
• MadryLab / DsDm
  ☆51Updated last year
• IBM / ColPret
  Efficient Scaling laws and collaborative pretraining.
  ☆19Updated 4 months ago
• Infini-AI-Lab / GRESO
  ☆73Updated 6 months ago
• osehmathias / lisa
  LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
  ☆36Updated last year
• holarissun / RewardModelingBeyondBradleyTerry
  official implementation of ICLR'2025 paper: Rethinking Bradley-Terry Models in Preference-based Reward Modeling: Foundations, Theory, and…
  ☆70Updated 9 months ago
• berlino / seq_icl
  ☆53Updated last year
• tilde-research / nsa-impl
  An efficient implementation of the NSA (Native Sparse Attention) kernel
  ☆128Updated 6 months ago
• HanseulJo / position-coupling
  Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure (NeurIPS 2024) + Arithmetic Transfor…
  ☆11Updated 2 months ago
• epfml / llm-optimizer-benchmark
  Benchmarking Optimizers for LLM Pretraining
  ☆48Updated 2 weeks ago
• ryoungj / BoLT
  Code for "Reasoning to Learn from Latent Thoughts"
  ☆124Updated 9 months ago
• socialfoundations / tttlm
  Test-time-training on nearest neighbors for large language models
  ☆49Updated last year
• VITA-Group / LoCoCo
  [ICML‘2024] "LoCoCo: Dropping In Convolutions for Long Context Compression", Ruisi Cai, Yuandong Tian, Zhangyang Wang, Beidi Chen
  ☆18Updated last year