zqOuO / GWTLinks
☆13Updated 8 months ago
Alternatives and similar repositories for GWT
Users that are interested in GWT are comparing it to the libraries listed below
Sorting:
- ☆35Updated 6 months ago
- ☆39Updated 2 weeks ago
- Work in progress.☆72Updated 2 months ago
- This repository contains code for the MicroAdam paper.☆19Updated 9 months ago
- ☆12Updated 6 months ago
- The evaluation framework for training-free sparse attention in LLMs☆93Updated 3 months ago
- ☆57Updated 11 months ago
- ☆85Updated last year
- ☆53Updated last year
- [NeurIPS 2024] Low rank memory efficient optimizer without SVD☆31Updated 2 months ago
- An extention to the GaLore paper, to perform Natural Gradient Descent in low rank subspace☆17Updated 11 months ago
- From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…☆48Updated 5 months ago
- SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)☆34Updated 10 months ago
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆83Updated 10 months ago
- ☆83Updated last year
- Here we will test various linear attention designs.☆62Updated last year
- Muon fsdp 2☆43Updated last month
- ☆32Updated last year
- A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.☆70Updated last year
- ☆42Updated 5 months ago
- Flash-Muon: An Efficient Implementation of Muon Optimizer☆185Updated 3 months ago
- Triton Implementation of HyperAttention Algorithm☆48Updated last year
- ☆64Updated 6 months ago
- ☆14Updated 11 months ago
- Official Pytorch Implementation of "The Curse of Depth in Large Language Models" by Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin,Yefen…☆56Updated last month
- ☆29Updated 10 months ago
- [ICLR 2025 & COLM 2025] Official PyTorch implementation of the Forgetting Transformer and Adaptive Computation Pruning☆130Updated last week
- Unofficial Implementation of Selective Attention Transformer☆17Updated 10 months ago
- Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)☆81Updated 2 years ago
- Pytorch implementation of "Oscillation-Reduced MXFP4 Training for Vision Transformers" on DeiT Model Pre-training☆28Updated 3 months ago