apoorvkh / academic-pretraining
$100K or 100 Days: Trade-offs when Pre-Training with Academic Resources
☆135Updated last week
Alternatives and similar repositories for academic-pretraining:
Users that are interested in academic-pretraining are comparing it to the libraries listed below
- Code to reproduce "Transformers Can Do Arithmetic with the Right Embeddings", McLeish et al (NeurIPS 2024)☆186Updated 9 months ago
- An implementation of PSGD Kron second-order optimizer for PyTorch☆83Updated last month
- ☆73Updated 7 months ago
- ☆124Updated this week
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆100Updated 4 months ago
- PyTorch library for Active Fine-Tuning☆61Updated last month
- A MAD laboratory to improve AI architecture designs 🧪☆107Updated 3 months ago
- OpenCoconut implements a latent reasoning paradigm where we generate thoughts before decoding.☆168Updated 2 months ago
- WIP☆93Updated 7 months ago
- ☆76Updated 8 months ago
- ☆104Updated 2 months ago
- Normalized Transformer (nGPT)☆162Updated 4 months ago
- ☆25Updated last year
- ☆79Updated 11 months ago
- An introduction to LLM Sampling☆77Updated 3 months ago
- Understand and test language model architectures on synthetic tasks.☆184Updated 2 weeks ago
- ☆53Updated last year
- supporting pytorch FSDP for optimizers☆79Updated 3 months ago
- Official implementation of "BERTs are Generative In-Context Learners"☆25Updated last week
- Focused on fast experimentation and simplicity☆69Updated 2 months ago
- [ICLR 2025] Official PyTorch Implementation of Gated Delta Networks: Improving Mamba2 with Delta Rule☆142Updated this week
- Collection of autoregressive model implementation☆83Updated last month
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆122Updated 11 months ago
- 🧱 Modula software package☆173Updated last week
- nanoGPT-like codebase for LLM training☆90Updated this week