NUS-HPC-AI-Lab / pytorch-lamb
PyTorch implementation of LAMB for ImageNet/ResNet-50 training
☆13Updated 3 years ago
Alternatives and similar repositories for pytorch-lamb:
Users that are interested in pytorch-lamb are comparing it to the libraries listed below
- Accuracy 77%. Large batch deep learning optimizer LARS for ImageNet with PyTorch and ResNet, using Horovod for distribution. Optional acc…☆38Updated 3 years ago
- ☆35Updated 3 years ago
- Towards Understanding Sharpness-Aware Minimization [ICML 2022]☆35Updated 2 years ago
- ☆40Updated 3 years ago
- Code release for Deep Incubation (https://arxiv.org/abs/2212.04129)☆90Updated 2 years ago
- Practical low-rank gradient compression for distributed optimization: https://arxiv.org/abs/1905.13727☆146Updated 6 months ago
- ☆31Updated last year
- Code for Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot☆42Updated 4 years ago
- Code for "Training Neural Networks with Fixed Sparse Masks" (NeurIPS 2021).☆58Updated 3 years ago
- Code for "Picking Winning Tickets Before Training by Preserving Gradient Flow" https://openreview.net/pdf?id=SkgsACVKPH☆103Updated 5 years ago
- Parameter Efficient Transfer Learning with Diff Pruning☆73Updated 4 years ago
- Code accompanying the NeurIPS 2020 paper: WoodFisher (Singh & Alistarh, 2020)☆50Updated 4 years ago
- Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)☆80Updated last year
- Block Sparse movement pruning☆79Updated 4 years ago
- ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training☆200Updated 2 years ago
- This package implements THOR: Transformer with Stochastic Experts.☆61Updated 3 years ago
- ☆42Updated 2 years ago
- [ICLR 2023] "Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!" Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen…☆28Updated last year
- ☆204Updated 2 years ago
- Preprint: Asymmetry in Low-Rank Adapters of Foundation Models☆36Updated last year
- Code for the paper: Why Transformers Need Adam: A Hessian Perspective☆57Updated last month
- Lightweight torch implementation of rigl, a sparse-to-sparse optimizer.☆56Updated 3 years ago
- [ICML 2021 Oral] "CATE: Computation-aware Neural Architecture Encoding with Transformers" by Shen Yan, Kaiqiang Song, Fei Liu, Mi Zhang☆19Updated 3 years ago
- [ICML 2021] "Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training" by Shiwei Liu, Lu Yin, De…☆45Updated last year
- Distributed K-FAC preconditioner for PyTorch☆85Updated this week
- Soft Threshold Weight Reparameterization for Learnable Sparsity☆89Updated 2 years ago
- Host CIFAR-10.2 Data Set☆13Updated 3 years ago
- Code associated with the paper **Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees**.☆28Updated 2 years ago
- Code and checkpoints of compressed networks for the paper titled "HYDRA: Pruning Adversarially Robust Neural Networks" (NeurIPS 2020) (ht…☆92Updated 2 years ago
- ☆19Updated 3 years ago