Alternatives and similar repositories for LayerNorm-Scaling:

Users that are interested in LayerNorm-Scaling are comparing it to the libraries listed below

• RobertCsordas / moeut
  ☆78Updated 8 months ago
• Infini-AI-Lab / S2FT
  ☆17Updated 3 months ago
• dangxingyu / rnn-icrag
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆26Updated last year
• PKU-ML / LongPPL
  Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"
  ☆53Updated last month
• formll / resolving-scaling-law-discrepancies
  ☆18Updated 9 months ago
• katiekang1998 / reasoning_generalization
  ☆31Updated 3 months ago
• TianjinYellow / SPAM-Optimizer
  ☆27Updated last month
• OpenSparseLLMs / Linearization
  ☆39Updated last month
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆54Updated 8 months ago
• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆52Updated last month
• sail-sg / SkyLadder
  The official repository for SkyLadder: Better and Faster Pretraining via Context Window Scheduling
  ☆29Updated last month
• kyleliang919 / Online-Subspace-Descent
  This repo is based on https://github.com/jiaweizzhao/GaLore
  ☆26Updated 7 months ago
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆71Updated 5 months ago
• ml-jku / EVA
  One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
  ☆39Updated 6 months ago
• kiddyboots216 / lottery-ticket-adaptation
  Lottery Ticket Adaptation
  ☆39Updated 5 months ago
• sjelassi / transformers_ssm_copy
  ☆31Updated last year
• kamanphoebe / Look-into-MoEs
  [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models
  ☆51Updated 2 months ago
• ldery / Bonsai
  Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"
  ☆27Updated last year
• BlinkDL / LinearAttentionArena
  Here we will test various linear attention designs.
  ☆60Updated last year
• hamishivi / automated-instruction-selection
  Exploration of automated dataset selection approaches at large scales.
  ☆39Updated last month
• NathanGodey / qfilters
  Repository for the Q-Filters method (https://arxiv.org/pdf/2503.02812)
  ☆30Updated last month
• Leiay / looped_transformer
  ☆25Updated last year
• VITA-Group / LoCoCo
  [ICML‘2024] "LoCoCo: Dropping In Convolutions for Long Context Compression", Ruisi Cai, Yuandong Tian, Zhangyang Wang, Beidi Chen
  ☆16Updated 7 months ago
• matchten / LoRA-Models-for-SAEs
  Code for reproducing our paper "Low Rank Adapting Models for Sparse Autoencoder Features"
  ☆10Updated 3 weeks ago
• ryoungj / BoLT
  Code for "Reasoning to Learn from Latent Thoughts"
  ☆91Updated 3 weeks ago
• IST-DASLab / RoSA
  Official implementation of the ICML 2024 paper RoSA (Robust Adaptation)
  ☆40Updated last year
• facebookresearch / MemoryMosaics
  Memory Mosaics are networks of associative memories working in concert to achieve a prediction task.
  ☆40Updated 2 months ago
• g-luo / task_vectors_are_cross_modal
  Official PyTorch Implementation for Task Vectors are Cross-Modal
  ☆22Updated 4 months ago
• thunlp / Modularity-Analysis
  Repo for ACL2023 Findings paper "Emergent Modularity in Pre-trained Transformers"
  ☆23Updated last year
• socialfoundations / tttlm
  Test-time-training on nearest neighbors for large language models
  ☆40Updated last year