ClashLuke / HeavyBall
Efficient optimizers
☆58Updated this week
Related projects ⓘ
Alternatives and complementary repositories for HeavyBall
- ☆121Updated this week
- Simple implementation of muP, based on Spectral Condition for Feature Learning. The implementation is SGD only, dont use it for Adam☆69Updated 3 months ago
- ☆31Updated 2 months ago
- ☆21Updated 4 months ago
- An implementation of PSGD Kron second-order optimizer for PyTorch☆15Updated this week
- Implementation of GateLoop Transformer in Pytorch and Jax☆86Updated 4 months ago
- WIP☆89Updated 2 months ago
- Muon optimizer for neural networks: >30% extra sample efficiency, <3% wallclock overhead☆69Updated this week
- ☆50Updated 10 months ago
- ☆72Updated 4 months ago
- Scalable neural net training via automatic normalization in the modular norm.☆119Updated 2 months ago
- Utilities for PyTorch distributed☆23Updated last year
- ☆18Updated last month
- ☆53Updated 9 months ago
- A JAX implementation of the continuous time formulation of Consistency Models☆83Updated last year
- Accelerated First Order Parallel Associative Scan☆162Updated 2 months ago
- Explorations into the proposal from the paper "Grokfast, Accelerated Grokking by Amplifying Slow Gradients"☆84Updated 2 months ago
- Implementation of the proposed Adam-atan2 from Google Deepmind in Pytorch☆94Updated 2 weeks ago
- ☆26Updated 6 months ago
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆112Updated 6 months ago
- Explorations into the recently proposed Taylor Series Linear Attention☆89Updated 2 months ago
- Experiment of using Tangent to autodiff triton☆71Updated 9 months ago
- ☆76Updated 5 months ago
- ☆46Updated last month
- ☆46Updated last month
- Just some miscellaneous utility functions / decorators / modules related to Pytorch and Accelerate to help speed up implementation of new…☆117Updated 3 months ago
- PyTorch half precision gemm lib w/ fused optional bias + optional relu/gelu☆38Updated 2 months ago
- σ-GPT: A New Approach to Autoregressive Models☆59Updated 2 months ago
- ☆76Updated 6 months ago
- Train vision models using JAX and 🤗 transformers☆95Updated 3 weeks ago