Alternatives and similar repositories for Active-Dormant-Attention

Users that are interested in Active-Dormant-Attention are comparing it to the libraries listed below

• fangyuan-ksgk / selective-attention-transformer
  Unofficial Implementation of Selective Attention Transformer
  ☆18Updated last year
• sjelassi / transformers_ssm_copy
  ☆35Updated last year
• ablghtianyi / ICL_Modular_Arithmetic
  ☆19Updated 9 months ago
• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆62Updated 5 months ago
• formll / resolving-scaling-law-discrepancies
  ☆20Updated last month
• alvin-zyl / CoLA
  Implementation of CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
  ☆25Updated 10 months ago
• andyjm3 / SLTrain
  SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)
  ☆38Updated last year
• alessiodevoto / l2compress
  Code for the EMNLP24 paper "A simple and effective L2 norm based method for KV Cache compression."
  ☆17Updated last year
• HanseulJo / position-coupling
  Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure (NeurIPS 2024) + Arithmetic Transfor…
  ☆11Updated 2 months ago
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆86Updated last year
• sail-sg / Attention-Sink
  [ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)
  ☆151Updated 5 months ago
• zyushun / hessian-spectrum
  Code for the paper: Why Transformers Need Adam: A Hessian Perspective
  ☆63Updated 9 months ago
• Infini-AI-Lab / Kinetics
  Kinetics: Rethinking Test-Time Scaling Laws
  ☆84Updated 5 months ago
• nasosger / MuToR
  [NeurIPS '25] Multi-Token Prediction Needs Registers
  ☆26Updated 2 weeks ago
• deep-spin / adasplash
  AdaSplash: Adaptive Sparse Flash Attention (aka Flash Entmax Attention)
  ☆31Updated 3 months ago
• VITA-Group / LoCoCo
  [ICML‘2024] "LoCoCo: Dropping In Convolutions for Long Context Compression", Ruisi Cai, Yuandong Tian, Zhangyang Wang, Beidi Chen
  ☆18Updated last year
• sustcsonglin / linear-attention-and-beyond-slides
  ☆101Updated 10 months ago
• PKU-ML / LongPPL
  Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"
  ☆107Updated 2 months ago
• leloykun / flash-attention-minimal
  Flash Attention in 300-500 lines of CUDA/C++
  ☆36Updated 4 months ago
• tilde-research / nsa-impl
  An efficient implementation of the NSA (Native Sparse Attention) kernel
  ☆127Updated 6 months ago
• Lucky-Lance / SPP
  [ICML 2024] SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
  ☆21Updated last year
• test-time-training / ttt-tk
  ☆43Updated last month
• TianjinYellow / SPAM-Optimizer
  ☆36Updated 9 months ago
• Leiay / looped_transformer
  ☆33Updated 2 years ago
• sail-sg / SkyLadder
  The official repository for SkyLadder: Better and Faster Pretraining via Context Window Scheduling
  ☆40Updated 2 months ago
• AkideLiu / MiniCache
  ☆10Updated last year
• VITA-Group / Junk_DNA_Hypothesis
  [ICML 2024] Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity; Lu Yin*, Ajay Jaiswal*, Shiwei Liu, So…
  ☆16Updated 8 months ago
• lmsdss / LayerNorm-Scaling
  Official Pytorch Implementation of "The Curse of Depth in Large Language Models" by Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin,Yefen…
  ☆64Updated 3 weeks ago
• epfml / llm-optimizer-benchmark
  Benchmarking Optimizers for LLM Pretraining
  ☆47Updated last week
• r-three / smear
  ☆30Updated 2 years ago