Alternatives and similar repositories for Active-Dormant-Attention

Users that are interested in Active-Dormant-Attention are comparing it to the libraries listed below

• andyjm3 / SLTrain
  SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)
  ☆38Updated last year
• sjelassi / transformers_ssm_copy
  ☆35Updated last year
• ablghtianyi / ICL_Modular_Arithmetic
  ☆19Updated 8 months ago
• formll / resolving-scaling-law-discrepancies
  ☆20Updated last month
• fangyuan-ksgk / selective-attention-transformer
  Unofficial Implementation of Selective Attention Transformer
  ☆18Updated last year
• alvin-zyl / CoLA
  Implementation of CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
  ☆24Updated 9 months ago
• Infini-AI-Lab / Kinetics
  Kinetics: Rethinking Test-Time Scaling Laws
  ☆84Updated 4 months ago
• PKU-ML / LongPPL
  Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"
  ☆105Updated last month
• sail-sg / Attention-Sink
  [ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)
  ☆143Updated 5 months ago
• zyushun / hessian-spectrum
  Code for the paper: Why Transformers Need Adam: A Hessian Perspective
  ☆63Updated 8 months ago
• alessiodevoto / l2compress
  Code for the EMNLP24 paper "A simple and effective L2 norm based method for KV Cache compression."
  ☆17Updated 11 months ago
• VITA-Group / LoCoCo
  [ICML‘2024] "LoCoCo: Dropping In Convolutions for Long Context Compression", Ruisi Cai, Yuandong Tian, Zhangyang Wang, Beidi Chen
  ☆18Updated last year
• Model-GLUE / Model-GLUE
  ☆18Updated last year
• deep-spin / adasplash
  AdaSplash: Adaptive Sparse Flash Attention (aka Flash Entmax Attention)
  ☆30Updated 2 months ago
• haotiansun14 / BBox-Adapter
  Lightweight Adapting for Black-Box Large Language Models
  ☆24Updated last year
• nasosger / MuToR
  [NeurIPS '25] Multi-Token Prediction Needs Registers
  ☆25Updated last week
• princeton-pli / what-makes-good-rm
  [NeurIPS 2025] What Makes a Reward Model a Good Teacher? An Optimization Perspective
  ☆40Updated 2 months ago
• bethgelab / sober-reasoning
  A Sober Look at Language Model Reasoning
  ☆89Updated 3 weeks ago
• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆61Updated 5 months ago
• Lucky-Lance / SPP
  [ICML 2024] SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
  ☆21Updated last year
• osehmathias / lisa
  LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
  ☆36Updated last year
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆85Updated last year
• tim-lawson / mlsae
  Multi-Layer Sparse Autoencoders (ICLR 2025)
  ☆27Updated 9 months ago
• sustcsonglin / linear-attention-and-beyond-slides
  ☆100Updated 9 months ago
• VITA-Group / Junk_DNA_Hypothesis
  [ICML 2024] Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity; Lu Yin*, Ajay Jaiswal*, Shiwei Liu, So…
  ☆16Updated 7 months ago
• MadryLab / DsDm
  ☆51Updated last year
• socialfoundations / tttlm
  Test-time-training on nearest neighbors for large language models
  ☆48Updated last year
• ryoungj / BoLT
  Code for "Reasoning to Learn from Latent Thoughts"
  ☆122Updated 8 months ago
• YefanZhou / TempBalance
  [NeurIPS 2023 Spotlight] Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training
  ☆36Updated 8 months ago
• Infini-AI-Lab / GRESO
  ☆69Updated 5 months ago