Alternatives and similar repositories for LayerNorm-Scaling

Users that are interested in LayerNorm-Scaling are comparing it to the libraries listed below

• RobertCsordas / moeut
  ☆91Updated last year
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆87Updated last year
• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆62Updated 6 months ago
• samsja / muon_fsdp_2
  Muon fsdp 2
  ☆48Updated 5 months ago
• sail-sg / SkyLadder
  The official repository for SkyLadder: Better and Faster Pretraining via Context Window Scheduling
  ☆41Updated 2 weeks ago
• lucidrains / coconut-pytorch
  Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch
  ☆181Updated 6 months ago
• jzhang38 / LongMamba
  Some preliminary explorations of Mamba's context scaling.
  ☆217Updated last year
• NathanGodey / qfilters
  Repository for the Q-Filters method (https://arxiv.org/pdf/2503.02812)
  ☆35Updated 10 months ago
• PKU-ML / LongPPL
  Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"
  ☆109Updated 3 months ago
• PiotrNawrot / sparse-frontier
  The evaluation framework for training-free sparse attention in LLMs
  ☆108Updated 3 months ago
• Infini-AI-Lab / Kinetics
  Kinetics: Rethinking Test-Time Scaling Laws
  ☆85Updated 6 months ago
• TianjinYellow / SPAM-Optimizer
  ☆36Updated 10 months ago
• locuslab / massive-activations
  Code accompanying the paper "Massive Activations in Large Language Models"
  ☆193Updated last year
• nasosger / MuToR
  [NeurIPS '25] Multi-Token Prediction Needs Registers
  ☆26Updated last month
• mlfoundations / scaling
  Language models scale reliably with over-training and on downstream tasks
  ☆100Updated last year
• Leiay / looped_transformer
  ☆33Updated 2 years ago
• kyleliang919 / Online-Subspace-Descent
  [NeurIPS 2024] Low rank memory efficient optimizer without SVD
  ☆32Updated 6 months ago
• tilde-research / nsa-impl
  An efficient implementation of the NSA (Native Sparse Attention) kernel
  ☆128Updated 6 months ago
• JeanKaddour / NoTrainNoGain
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆81Updated 2 years ago
• fangyuan-ksgk / selective-attention-transformer
  Unofficial Implementation of Selective Attention Transformer
  ☆19Updated last year
• Infini-AI-Lab / gsm_infinite
  ☆61Updated 7 months ago
• allenai / signal-and-noise
  Measuring the Signal to Noise Ratio in Language Model Evaluation
  ☆28Updated 4 months ago
• r-three / smear
  ☆30Updated 2 years ago
• kamanphoebe / Look-into-MoEs
  [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models
  ☆57Updated 11 months ago
• ml-jku / EVA
  One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
  ☆46Updated 2 months ago
• smonsays / hypernetwork-attention
  Official code for the paper "Attention as a Hypernetwork"
  ☆47Updated last year
• MadryLab / DsDm
  ☆51Updated last year
• berlino / seq_icl
  ☆53Updated last year
• mengxiayu / LLMSuperWeight
  Code for studying the super weight in LLM
  ☆120Updated last year
• minyoungg / LTE
  ☆71Updated last year