lmsdss / LayerNorm-ScalingLinks
Official Pytorch Implementation of "The Curse of Depth in Large Language Models" by Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin,Yefeng Zheng, Shiwei Liu
☆55Updated last month
Alternatives and similar repositories for LayerNorm-Scaling
Users that are interested in LayerNorm-Scaling are comparing it to the libraries listed below
Sorting:
- ☆85Updated last year
- Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"☆97Updated last month
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆82Updated 10 months ago
- ☆34Updated 5 months ago
- ☆102Updated 11 months ago
- The evaluation framework for training-free sparse attention in LLMs☆91Updated 2 months ago
- Some preliminary explorations of Mamba's context scaling.☆217Updated last year
- ☆11Updated 5 months ago
- [NeurIPS 2024] Low rank memory efficient optimizer without SVD☆30Updated 2 months ago
- The official repository for SkyLadder: Better and Faster Pretraining via Context Window Scheduling☆33Updated last week
- Official PyTorch Implementation of the Longhorn Deep State Space Model☆54Updated 8 months ago
- Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…☆114Updated 11 months ago
- Kinetics: Rethinking Test-Time Scaling Laws☆79Updated last month
- ☆14Updated 11 months ago
- Repository for the Q-Filters method (https://arxiv.org/pdf/2503.02812)☆34Updated 5 months ago
- Stick-breaking attention☆59Updated 2 months ago
- Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch☆179Updated 2 months ago
- Code accompanying the paper "Massive Activations in Large Language Models"☆176Updated last year
- ☆19Updated 7 months ago
- ☆98Updated last year
- From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…☆47Updated 4 months ago
- An efficient implementation of the NSA (Native Sparse Attention) kernel☆114Updated 2 months ago
- Muon fsdp 2☆42Updated 3 weeks ago
- Reinforcing General Reasoning without Verifiers☆80Updated 2 months ago
- Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"☆28Updated last year
- ☆80Updated 6 months ago
- [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"☆92Updated 2 months ago
- ☆20Updated last year
- Code for "Reasoning to Learn from Latent Thoughts"☆116Updated 5 months ago
- Language models scale reliably with over-training and on downstream tasks☆98Updated last year