epfml / schedules-and-scalingLinks
Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
☆84Updated 11 months ago
Alternatives and similar repositories for schedules-and-scaling
Users that are interested in schedules-and-scaling are comparing it to the libraries listed below
Sorting:
- Language models scale reliably with over-training and on downstream tasks☆100Updated last year
- ☆86Updated last year
- ☆53Updated last year
- Using FlexAttention to compute attention with different masking patterns☆47Updated last year
- Simple and efficient pytorch-native transformer training and inference (batched)☆78Updated last year
- ☆101Updated last year
- A toolkit for scaling law research ⚖☆52Updated 8 months ago
- Official PyTorch implementation and models for paper "Diffusion Beats Autoregressive in Data-Constrained Settings". We find diffusion mod…☆101Updated last month
- Gemstones: A Model Suite for Multi-Faceted Scaling Laws (NeurIPS 2025)☆29Updated 3 weeks ago
- ☆107Updated last year
- nanoGPT-like codebase for LLM training☆107Updated 5 months ago
- Stick-breaking attention☆61Updated 3 months ago
- Code and Configs for Asynchronous RLHF: Faster and More Efficient RL for Language Models☆64Updated 5 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆60Updated last year
- ☆33Updated 9 months ago
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆166Updated 3 months ago
- ☆83Updated last year
- [NeurIPS 2024] Goldfish Loss: Mitigating Memorization in Generative LLMs☆92Updated 11 months ago
- ☆20Updated last year
- A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.☆69Updated last year
- Triton Implementation of HyperAttention Algorithm☆48Updated last year
- Official code repo for paper "Great Memory, Shallow Reasoning: Limits of kNN-LMs"☆24Updated 5 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆27Updated last year
- Exploration of automated dataset selection approaches at large scales.☆47Updated 7 months ago
- The official repository for SkyLadder: Better and Faster Pretraining via Context Window Scheduling☆35Updated last week
- The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.☆91Updated 3 months ago
- [ICLR 2025] Monet: Mixture of Monosemantic Experts for Transformers☆73Updated 4 months ago
- ☆33Updated last year
- Here we will test various linear attention designs.☆61Updated last year
- ☆32Updated last year