pytorch-labs / attention-gym
Helpful tools and examples for working with flex-attention
☆460Updated 2 weeks ago
Related projects ⓘ
Alternatives and complementary repositories for attention-gym
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆474Updated 2 weeks ago
- A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.☆473Updated 2 weeks ago
- This repository contains the experimental PyTorch native float8 training UX☆211Updated 3 months ago
- Annotated version of the Mamba paper☆455Updated 8 months ago
- Transformers with Arbitrarily Large Context☆637Updated 2 months ago
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆190Updated 2 weeks ago
- FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores☆277Updated last month
- Ring attention implementation with flash attention☆578Updated this week
- Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793☆321Updated last week
- Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch☆291Updated 4 months ago
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆212Updated 2 months ago
- ☆132Updated last year
- Triton-based implementation of Sparse Mixture of Experts.☆185Updated 3 weeks ago
- For optimization algorithm research and development.☆408Updated this week
- Efficient implementations of state-of-the-art linear attention models in Pytorch and Triton☆1,320Updated this week
- Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAI☆234Updated this week
- Some preliminary explorations of Mamba's context scaling.☆190Updated 9 months ago
- Accelerated First Order Parallel Associative Scan☆162Updated 2 months ago
- Pipeline Parallelism for PyTorch☆725Updated 2 months ago
- Building blocks for foundation models.☆385Updated 10 months ago
- Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch☆242Updated 6 months ago
- ☆227Updated 2 months ago
- Explorations into some recent techniques surrounding speculative decoding☆209Updated last year
- Understand and test language model architectures on synthetic tasks.☆161Updated 6 months ago
- Cataloging released Triton kernels.☆132Updated 2 months ago
- [ICML 2024] CLLMs: Consistency Large Language Models☆350Updated 3 months ago
- Microsoft Automatic Mixed Precision Library☆522Updated last month
- A repository for log-time feedforward networks☆216Updated 7 months ago
- Fast Matrix Multiplications for Lookup Table-Quantized LLMs☆183Updated last month
- A repository for research on medium sized language models.☆479Updated this week