meta-pytorch / attention-gymLinks
Helpful tools and examples for working with flex-attention
☆1,116Updated 2 weeks ago
Alternatives and similar repositories for attention-gym
Users that are interested in attention-gym are comparing it to the libraries listed below
Sorting:
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆549Updated 8 months ago
- 🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"☆964Updated 10 months ago
- A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.☆596Updated 5 months ago
- ☆579Updated 4 months ago
- Large Context Attention☆764Updated 3 months ago
- Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper☆796Updated 5 months ago
- Pipeline Parallelism for PyTorch☆785Updated last year
- Muon is an optimizer for hidden layers in neural networks☆2,242Updated last week
- Ring attention implementation with flash attention☆973Updated 4 months ago
- LLM KV cache compression made easy☆866Updated this week
- [ICLR 2025] Official PyTorch Implementation of Gated Delta Networks: Improving Mamba2 with Delta Rule☆433Updated 4 months ago
- Annotated version of the Mamba paper☆495Updated last year
- A Quirky Assortment of CuTe Kernels☆772Updated this week
- Scalable and Performant Data Loading☆362Updated last week
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆474Updated 2 weeks ago
- Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch☆796Updated 6 months ago
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆280Updated 2 months ago
- [ICLR2025 Spotlight🔥] Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters☆584Updated 11 months ago
- Load compute kernels from the Hub☆389Updated this week
- A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Data Training☆626Updated this week
- A Python-embedded DSL that makes it easy to write fast, scalable ML kernels with minimal boilerplate.☆732Updated this week
- Tutel MoE: Optimized Mixture-of-Experts Library, Support GptOss/DeepSeek/Kimi-K2/Qwen3 using FP8/NVFP4/MXFP4☆960Updated last month
- Flash Attention in ~100 lines of CUDA (forward pass only)☆1,058Updated last year
- For optimization algorithm research and development.☆558Updated 2 weeks ago
- Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793☆451Updated 8 months ago
- ☆952Updated 2 months ago
- Accelerating MoE with IO and Tile-aware Optimizations☆563Updated last week
- depyf is a tool to help you understand and adapt to PyTorch compiler torch.compile.☆782Updated 3 months ago
- Best practices & guides on how to write distributed pytorch training code☆571Updated 3 months ago
- H-Net: Hierarchical Network with Dynamic Chunking☆807Updated 2 months ago