microsoft / dionLinks
Dion optimizer algorithm
☆318Updated last week
Alternatives and similar repositories for dion
Users that are interested in dion are comparing it to the libraries listed below
Sorting:
- Simple & Scalable Pretraining for Neural Architecture Research☆290Updated last week
- Load compute kernels from the Hub☆258Updated this week
- ☆275Updated last year
- PyTorch Single Controller☆374Updated this week
- 🧱 Modula software package☆231Updated 2 weeks ago
- Normalized Transformer (nGPT)☆187Updated 9 months ago
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆156Updated 2 months ago
- FlexAttention based, minimal vllm-style inference engine for fast Gemma 2 inference.☆261Updated 3 weeks ago
- supporting pytorch FSDP for optimizers☆84Updated 8 months ago
- Minimal yet performant LLM examples in pure JAX☆150Updated last week
- Efficient optimizers☆256Updated 3 weeks ago
- ☆384Updated this week
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆389Updated 2 weeks ago
- Home for "How To Scale Your Model", a short blog-style textbook about scaling LLMs on TPUs☆562Updated this week
- ☆214Updated 6 months ago
- Scalable and Performant Data Loading☆295Updated this week
- DeMo: Decoupled Momentum Optimization☆190Updated 8 months ago
- ☆87Updated last year
- A MAD laboratory to improve AI architecture designs 🧪☆127Updated 8 months ago
- Getting crystal-like representations with harmonic loss☆194Updated 4 months ago
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆245Updated 7 months ago
- For optimization algorithm research and development.☆530Updated last week
- An implementation of PSGD Kron second-order optimizer for PyTorch☆96Updated last month
- ☆207Updated 8 months ago
- A repository to unravel the language of GPUs, making their kernel conversations easy to understand☆191Updated 3 months ago
- seqax = sequence modeling + JAX☆166Updated last month
- A JAX-native LLM Post-Training Library☆128Updated this week
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆131Updated last year
- CIFAR-10 speedruns: 94% in 2.6 seconds and 96% in 27 seconds☆284Updated last month
- Accelerated First Order Parallel Associative Scan☆187Updated last year