apoorvkh / academic-pretraining
$100K or 100 Days: Trade-offs when Pre-Training with Academic Resources
☆95Updated 2 weeks ago
Related projects ⓘ
Alternatives and complementary repositories for academic-pretraining
- Code to reproduce "Transformers Can Do Arithmetic with the Right Embeddings", McLeish et al (NeurIPS 2024)☆178Updated 5 months ago
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆84Updated last week
- ☆101Updated 3 months ago
- Normalized Transformer (nGPT)☆66Updated this week
- ☆73Updated 4 months ago
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆113Updated 7 months ago
- Collection of autoregressive model implementation☆67Updated this week
- code for training & evaluating Contextual Document Embedding models☆117Updated this week
- ☆62Updated 3 months ago
- A MAD laboratory to improve AI architecture designs 🧪☆95Updated 6 months ago
- σ-GPT: A New Approach to Autoregressive Models☆59Updated 3 months ago
- WIP☆89Updated 3 months ago
- ☆53Updated 10 months ago
- Muon optimizer for neural networks: >30% extra sample efficiency, <3% wallclock overhead☆109Updated last week
- Official implementation of MAIA, A Multimodal Automated Interpretability Agent☆62Updated 3 months ago
- Scalable neural net training via automatic normalization in the modular norm.☆121Updated 3 months ago
- Implementation of the Llama architecture with RLHF + Q-learning☆157Updated 10 months ago
- ☆128Updated this week
- ☆107Updated this week
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆104Updated last month
- A puzzle to learn about prompting☆121Updated last year
- One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation☆29Updated last month
- Code for reproducing our paper "Not All Language Model Features Are Linear"☆61Updated last week
- [NeurIPS 2024] Goldfish Loss: Mitigating Memorization in Generative LLMs☆74Updated this week
- Language models scale reliably with over-training and on downstream tasks☆94Updated 7 months ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆46Updated 2 months ago
- Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…☆78Updated 2 months ago
- ☆76Updated 7 months ago
- ☆54Updated last month
- An introduction to LLM Sampling☆64Updated last week