foundation-model-stack / fms-fsdpLinks
🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash attention v2.
☆260Updated last month
Alternatives and similar repositories for fms-fsdp
Users that are interested in fms-fsdp are comparing it to the libraries listed below
Sorting:
- 🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.☆208Updated last week
- This repository contains the experimental PyTorch native float8 training UX☆224Updated last year
- Triton-based implementation of Sparse Mixture of Experts.☆233Updated 8 months ago
- Load compute kernels from the Hub☆244Updated this week
- Applied AI experiments and examples for PyTorch☆290Updated 2 months ago
- ring-attention experiments☆149Updated 10 months ago
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆536Updated 3 months ago
- Fast low-bit matmul kernels in Triton☆349Updated this week
- ☆211Updated 6 months ago
- Large Context Attention☆727Updated 7 months ago
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆383Updated last week
- Scalable toolkit for efficient model reinforcement☆626Updated this week
- Efficient LLM Inference over Long Sequences☆389Updated last month
- A Quirky Assortment of CuTe Kernels☆407Updated this week
- Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training☆214Updated last year
- Explorations into some recent techniques surrounding speculative decoding☆282Updated 8 months ago
- LLM KV cache compression made easy☆586Updated this week
- Cataloging released Triton kernels.☆252Updated 7 months ago
- ☆118Updated last year
- A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.☆568Updated last week
- Megatron's multi-modal data loader☆237Updated last week
- ☆232Updated this week
- ☆195Updated 3 months ago
- Flash-Muon: An Efficient Implementation of Muon Optimizer☆160Updated 2 months ago
- Collection of kernels written in Triton language☆145Updated 4 months ago
- 🔥 A minimal training framework for scaling FLA models☆228Updated last week
- ☆123Updated 2 months ago
- ☆514Updated 3 weeks ago
- kernels, of the mega variety☆472Updated 2 months ago
- ☆148Updated 2 years ago