epfml / schedules-and-scalingLinks
Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
☆77Updated 9 months ago
Alternatives and similar repositories for schedules-and-scaling
Users that are interested in schedules-and-scaling are comparing it to the libraries listed below
Sorting:
- Language models scale reliably with over-training and on downstream tasks☆97Updated last year
- The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.☆82Updated 2 weeks ago
- ☆82Updated 11 months ago
- ☆53Updated last year
- nanoGPT-like codebase for LLM training☆102Updated 2 months ago
- Stick-breaking attention☆58Updated last month
- Using FlexAttention to compute attention with different masking patterns☆44Updated 10 months ago
- Simple and efficient pytorch-native transformer training and inference (batched)☆78Updated last year
- Triton Implementation of HyperAttention Algorithm☆48Updated last year
- ☆88Updated last year
- Exploration of automated dataset selection approaches at large scales.☆47Updated 4 months ago
- ☆27Updated 5 months ago
- ☆37Updated last year
- ☆49Updated last year
- ☆81Updated last year
- Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)☆80Updated last year
- ☆53Updated 9 months ago
- ☆101Updated 10 months ago
- ☆33Updated last year
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆61Updated 9 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆27Updated last year
- ☆34Updated 6 months ago
- Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"☆92Updated last week
- Understand and test language model architectures on synthetic tasks.☆221Updated 2 weeks ago
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆149Updated last month
- ☆20Updated last year
- A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.☆70Updated last year
- ☆29Updated last year
- The evaluation framework for training-free sparse attention in LLMs☆86Updated last month
- Code and Configs for Asynchronous RLHF: Faster and More Efficient RL for Language Models☆59Updated 3 months ago