lucidrains / lion-pytorchLinks
🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch
☆2,164Updated 10 months ago
Alternatives and similar repositories for lion-pytorch
Users that are interested in lion-pytorch are comparing it to the libraries listed below
Sorting:
- maximal update parametrization (µP)☆1,605Updated last year
- The official implementation of “Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”☆971Updated last year
- Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models☆798Updated 3 months ago
- D-Adaptation for SGD, Adam and AdaGrad☆525Updated 8 months ago
- Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch☆753Updated 2 months ago
- torchview: visualize pytorch models☆990Updated 4 months ago
- Schedule-Free Optimization in PyTorch☆2,215Updated 4 months ago
- Foundation Architecture for (M)LLMs