EleutherAI / nanoGPT-mupLinks
The simplest, fastest repository for training/finetuning medium-sized GPTs.
β164Updated 3 months ago
Alternatives and similar repositories for nanoGPT-mup
Users that are interested in nanoGPT-mup are comparing it to the libraries listed below
Sorting:
- A MAD laboratory to improve AI architecture designs π§ͺβ129Updated 9 months ago
- Understand and test language model architectures on synthetic tasks.β226Updated last week
- nanoGPT-like codebase for LLM trainingβ107Updated 4 months ago
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT trainingβ132Updated last year
- β89Updated last year
- supporting pytorch FSDP for optimizersβ84Updated 9 months ago
- Code to reproduce "Transformers Can Do Arithmetic with the Right Embeddings", McLeish et al (NeurIPS 2024)β193Updated last year
- β53Updated last year
- Normalized Transformer (nGPT)β191Updated 10 months ago
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"β84Updated 11 months ago
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"β240Updated 3 months ago
- A set of Python scripts that makes your experience on TPU betterβ54Updated 2 weeks ago
- Language models scale reliably with over-training and on downstream tasksβ100Updated last year
- β102Updated 2 months ago
- Implementation of π₯₯ Coconut, Chain of Continuous Thought, in Pytorchβ180Updated 3 months ago
- β58Updated 11 months ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clustersβ130Updated 10 months ago
- π§± Modula software packageβ277Updated last month
- β142Updated 3 weeks ago
- WIPβ93Updated last year
- Minimal but scalable implementation of large language models in JAXβ35Updated last month
- πSmall Batch Size Training for Language Modelsβ62Updated last week
- β166Updated 2 years ago
- β82Updated last year
- A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.β70Updated last year
- β281Updated last year
- β215Updated 10 months ago
- some common Huggingface transformers in maximal update parametrization (Β΅P)β82Updated 3 years ago
- β196Updated last month
- Accelerated First Order Parallel Associative Scanβ188Updated last year