XuezheMax / apolloLinks
Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization
☆183Updated 3 years ago
Alternatives and similar repositories for apollo
Users that are interested in apollo are comparing it to the libraries listed below
Sorting:
- pytorch implement of Lookahead Optimizer☆190Updated 3 years ago
- Official PyTorch Repo for "ReZero is All You Need: Fast Convergence at Large Depth"☆408Updated 11 months ago
- Accelerate training by storing parameters in one contiguous chunk of memory.☆291Updated 4 years ago
- Implementation and experiments for AdamW on Pytorch☆94Updated 5 years ago
- lookahead optimizer (Lookahead Optimizer: k steps forward, 1 step back) for pytorch☆337Updated 5 years ago
- Implementation of Sparsemax activation in Pytorch☆160Updated 5 years ago
- Loss and accuracy go opposite ways...right?☆94Updated 5 years ago
- Code for the ICML'20 paper "Improving Transformer Optimization Through Better Initialization"☆89Updated 4 years ago
- The implementation of "Self-Supervised Generalisation with Meta Auxiliary Learning" [NeurIPS 2019].☆175Updated 3 years ago
- Sinkhorn Transformer - Practical implementation of Sparse Sinkhorn Attention☆264Updated 3 years ago
- ☆165Updated 6 years ago
- Pytorch Implementation of Neural Architecture Optimization☆113Updated 4 years ago
- [ICML 2020] code for "PowerNorm: Rethinking Batch Normalization in Transformers" https://arxiv.org/abs/2003.07845☆120Updated 4 years ago
- Implementation of https://arxiv.org/abs/1904.00962☆376Updated 4 years ago
- ☆84Updated 4 years ago
- Pytorch implementation of the hamburger module from the ICLR 2021 paper "Is Attention Better Than Matrix Decomposition"☆99Updated 4 years ago
- [ICML 2021 Oral] We show pure attention suffers rank collapse, and how different mechanisms combat it.☆165Updated 4 years ago
- A Re-implementation of Fixed-update Initialization☆153Updated 6 years ago
- ☆182Updated 2 years ago
- [ICML 2020] code for the flooding regularizer proposed in "Do We Need Zero Training Loss After Achieving Zero Training Error?"☆92Updated 2 years ago
- AdaX: Adaptive Gradient Descent with Exponential Long Term Momery☆34Updated 5 years ago
- Experimental ground for optimizing memory of pytorch models☆366Updated 7 years ago
- AutoTorch, A HPO Toolkit☆60Updated 5 years ago
- Implementing SYNTHESIZER: Rethinking Self-Attention in Transformer Models using Pytorch☆70Updated 5 years ago
- [ECCV 2020] DADA: Differentiable Automatic Data Augmentation☆190Updated last year
- PyTorch Examples repo for "ReZero is All You Need: Fast Convergence at Large Depth"☆62Updated 11 months ago
- Transformer with Untied Positional Encoding (TUPE). Code of paper "Rethinking Positional Encoding in Language Pre-training". Improve exis…☆251Updated 3 years ago
- PyTorch Code for "Evaluating the search phase of Neural Architecture Search" @ ICLR 2020☆50Updated 5 years ago
- A Pytorch implementation of "LegoNet: Efficient Convolutional Neural Networks with Lego Filters" (ICML 2019).☆140Updated 4 years ago
- Package of Optimizer implemented with PyTorch .☆66Updated 5 years ago