Alternatives and similar repositories for LayerNorm-Scaling

Users that are interested in LayerNorm-Scaling are comparing it to the libraries listed below

• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆85Updated last year
• RobertCsordas / moeut
  ☆88Updated last year
• TianjinYellow / SPAM-Optimizer
  ☆36Updated 8 months ago
• kyleliang919 / Online-Subspace-Descent
  [NeurIPS 2024] Low rank memory efficient optimizer without SVD
  ☆30Updated 4 months ago
• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆61Updated 4 months ago
• lucidrains / coconut-pytorch
  Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch
  ☆180Updated 5 months ago
• samsja / muon_fsdp_2
  Muon fsdp 2
  ☆45Updated 3 months ago
• sail-sg / SkyLadder
  The official repository for SkyLadder: Better and Faster Pretraining via Context Window Scheduling
  ☆40Updated last month
• VITA-Group / WeLore
  [ICML 2025] From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories and Applications
  ☆51Updated 3 weeks ago
• PKU-ML / LongPPL
  Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"
  ☆105Updated last month
• NathanGodey / qfilters
  Repository for the Q-Filters method (https://arxiv.org/pdf/2503.02812)
  ☆35Updated 8 months ago
• wang-kee / LiNeS
  Official repository of "LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging"
  ☆31Updated last year
• berlino / seq_icl
  ☆53Updated last year
• PiotrNawrot / sparse-frontier
  The evaluation framework for training-free sparse attention in LLMs
  ☆104Updated last month
• jzhang38 / LongMamba
  Some preliminary explorations of Mamba's context scaling.
  ☆217Updated last year
• deep-spin / adasplash
  AdaSplash: Adaptive Sparse Flash Attention (aka Flash Entmax Attention)
  ☆30Updated last month
• minyoungg / LTE
  ☆69Updated last year
• ScalingIntelligence / large_language_monkeys
  ☆109Updated last year
• nasosger / MuToR
  [NeurIPS '25] Multi-Token Prediction Needs Registers
  ☆24Updated 2 months ago
• Infini-AI-Lab / S2FT
  ☆19Updated 10 months ago
• goombalab / phi-mamba
  Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…
  ☆116Updated last year
• Anonymous1252022 / Megatron-DeepSpeed
  ☆14Updated last year
• Infini-AI-Lab / Kinetics
  Kinetics: Rethinking Test-Time Scaling Laws
  ☆82Updated 4 months ago
• Leiay / looped_transformer
  ☆33Updated last year
• dmis-lab / Monet
  [ICLR 2025] Monet: Mixture of Monosemantic Experts for Transformers
  ☆73Updated 5 months ago
• fangyuan-ksgk / selective-attention-transformer
  Unofficial Implementation of Selective Attention Transformer
  ☆17Updated last year
• formll / resolving-scaling-law-discrepancies
  ☆20Updated 3 weeks ago
• allenai / signal-and-noise
  Measuring the Signal to Noise Ratio in Language Model Evaluation
  ☆25Updated 3 months ago
• zqOuO / GWT
  ☆13Updated 10 months ago
• LIONS-EPFL / scion
  ☆47Updated last month