EleutherAI / nanoGPT-mup
The simplest, fastest repository for training/finetuning medium-sized GPTs.
☆84Updated this week
Related projects ⓘ
Alternatives and complementary repositories for nanoGPT-mup
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆113Updated 7 months ago
- ☆73Updated 4 months ago
- A MAD laboratory to improve AI architecture designs 🧪☆95Updated 6 months ago
- Understand and test language model architectures on synthetic tasks.☆163Updated 6 months ago
- Normalized Transformer (nGPT)☆87Updated this week
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆104Updated 2 months ago
- Experiments for efforts to train a new and improved t5☆76Updated 7 months ago
- Muon optimizer for neural networks: >30% extra sample efficiency, <3% wallclock overhead