NUS-HPC-AI-Lab / pytorch-lambLinks
PyTorch implementation of LAMB for ImageNet/ResNet-50 training
☆13Updated 4 years ago
Alternatives and similar repositories for pytorch-lamb
Users that are interested in pytorch-lamb are comparing it to the libraries listed below
Sorting:
- Accuracy 77%. Large batch deep learning optimizer LARS for ImageNet with PyTorch and ResNet, using Horovod for distribution. Optional acc…☆38Updated 4 years ago
- ☆222Updated 2 years ago
- Code for "Training Neural Networks with Fixed Sparse Masks" (NeurIPS 2021).☆59Updated 4 years ago
- Code accompanying the NeurIPS 2020 paper: WoodFisher (Singh & Alistarh, 2020)☆53Updated 4 years ago
- Practical low-rank gradient compression for distributed optimization: https://arxiv.org/abs/1905.13727☆149Updated last year
- ☆35Updated 4 years ago
- ☆43Updated 3 years ago
- Code for the paper: Why Transformers Need Adam: A Hessian Perspective☆63Updated 10 months ago
- Train ImageNet *fast* in 500 lines of code with FFCV☆149Updated last year
- Lightweight torch implementation of rigl, a sparse-to-sparse optimizer.☆60Updated 4 years ago
- Code for "Picking Winning Tickets Before Training by Preserving Gradient Flow" https://openreview.net/pdf?id=SkgsACVKPH☆105Updated 5 years ago
- [ICLR 2023] "Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!" Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen…☆28Updated 2 years ago
- Efficient reference implementations of the static & dynamic M-FAC algorithms (for pruning and optimization)☆17Updated 3 years ago
- Training vision models with full-batch gradient descent and regularization☆39Updated 2 years ago
- Distributed K-FAC preconditioner for PyTorch☆94Updated last week
- Understanding Training Dynamics of Deep ReLU Networks☆306Updated 3 months ago
- [IJCAI'22 Survey] Recent Advances on Neural Network Pruning at Initialization.☆59Updated 2 years ago
- ☆228Updated last year
- Mode Connectivity and Fast Geometric Ensembles in PyTorch☆283Updated 3 years ago
- ☆41Updated 4 years ago
- PyTorch repository for ICLR 2022 paper (GSAM) which improves generalization (e.g. +3.8% top-1 accuracy on ImageNet with ViT-B/32)☆146Updated 3 years ago
- Reproducing RigL (ICML 2020) as a part of ML Reproducibility Challenge 2020☆29Updated 4 years ago
- Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)☆81Updated 2 years ago
- Code associated with the paper **Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees**.☆28Updated 2 years ago
- Simple CIFAR10 ResNet example with JAX.☆23Updated 4 years ago
- ☆73Updated last year
- Soft Threshold Weight Reparameterization for Learnable Sparsity☆91Updated 2 years ago
- Code for Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot☆42Updated 5 years ago
- [ICML 2021] "Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training" by Shiwei Liu, Lu Yin, De…☆45Updated 2 years ago
- The implementation for MLSys 2023 paper: "Cuttlefish: Low-rank Model Training without All The Tuning"☆45Updated 2 years ago