pytorch-labs / attention-gym
Helpful tools and examples for working with flex-attention
☆469Updated 3 weeks ago
Related projects ⓘ
Alternatives and complementary repositories for attention-gym
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆476Updated 3 weeks ago
- Annotated version of the Mamba paper☆457Updated 8 months ago
- A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.☆483Updated 3 weeks ago
- This repository contains the experimental PyTorch native float8 training UX☆211Updated 3 months ago
- Transformers with Arbitrarily Large Context☆641Updated 3 months ago
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆193Updated this week
- Ring attention implementation with flash attention☆585Updated last week
- Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch☆293Updated 5 months ago
- FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores☆281Updated last month
- Triton-based implementation of Sparse Mixture of Experts.☆185Updated last month
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆214Updated 3 months ago
- Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793☆328Updated 3 weeks ago
- Pipeline Parallelism for PyTorch☆726Updated 2 months ago
- ☆132Updated last year
- Understand and test language model architectures on synthetic tasks.☆162Updated 6 months ago
- Building blocks for foundation models.☆394Updated 10 months ago
- Some preliminary explorations of Mamba's context scaling.☆191Updated 9 months ago
- Efficient implementations of state-of-the-art linear attention models in Pytorch and Triton☆1,339Updated this week
- ☆228Updated 2 months ago
- Accelerated First Order Parallel Associative Scan☆163Updated 3 months ago
- Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters☆335Updated last week
- Cataloging released Triton kernels.☆134Updated 2 months ago
- Applied AI experiments and examples for PyTorch☆166Updated 2 weeks ago
- Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch☆246Updated 6 months ago
- Microsoft Automatic Mixed Precision Library☆525Updated last month
- Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch☆571Updated last week
- For optimization algorithm research and development.☆449Updated this week
- 🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.☆165Updated this week
- [ICML 2024] CLLMs: Consistency Large Language Models☆353Updated this week
- Explorations into some recent techniques surrounding speculative decoding☆211Updated last year