ethansmith2000 / fsdp_optimizersLinks
supporting pytorch FSDP for optimizers
☆83Updated 10 months ago
Alternatives and similar repositories for fsdp_optimizers
Users that are interested in fsdp_optimizers are comparing it to the libraries listed below
Sorting:
- ☆91Updated last year
- Efficient optimizers☆274Updated last week
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆132Updated last year
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆166Updated 3 months ago
- Supporting code for the blog post on modular manifolds.☆86Updated 3 weeks ago
- WIP☆93Updated last year
- Simple implementation of muP, based on Spectral Condition for Feature Learning. The implementation is SGD only, dont use it for Adam☆85Updated last year
- ☆53Updated last year
- ☆217Updated 10 months ago
- A library for unit scaling in PyTorch☆130Updated 3 months ago
- Accelerated First Order Parallel Associative Scan☆189Updated last year
- research impl of Native Sparse Attention (2502.11089)☆62Updated 8 months ago
- DeMo: Decoupled Momentum Optimization☆194Updated 10 months ago
- ☆58Updated last year
- A MAD laboratory to improve AI architecture designs 🧪☆131Updated 10 months ago
- 🧱 Modula software package☆287Updated 2 months ago
- ☆30Updated 10 months ago
- 📄Small Batch Size Training for Language Models☆63Updated 2 weeks ago
- Fast, Modern, and Low Precision PyTorch Optimizers☆116Updated last month
- Understand and test language model architectures on synthetic tasks.☆233Updated 3 weeks ago
- Experiment of using Tangent to autodiff triton☆80Updated last year
- Tiny re-implementation of MDM in style of LLaDA and nano-gpt speedrun☆56Updated 7 months ago
- seqax = sequence modeling + JAX☆167Updated 3 months ago
- ☆67Updated 11 months ago
- nanoGPT-like codebase for LLM training☆107Updated 5 months ago
- ☆121Updated last year
- ☆34Updated last year
- Focused on fast experimentation and simplicity☆75Updated 9 months ago
- An implementation of PSGD Kron second-order optimizer for PyTorch☆96Updated 2 months ago
- ☆53Updated last year