lucidrains / ring-attention-pytorch
Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
☆510Updated 5 months ago
Alternatives and similar repositories for ring-attention-pytorch:
Users that are interested in ring-attention-pytorch are comparing it to the libraries listed below
- Large Context Attention☆700Updated 2 months ago
- Helpful tools and examples for working with flex-attention☆715Updated last week
- Ring attention implementation with flash attention☆734Updated last week
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆238Updated this week
- A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.☆529Updated last month
- Muon optimizer: +>30% sample efficiency with <3% wallclock overhead☆560Updated 3 weeks ago
- LLM KV cache compression made easy☆454Updated 3 weeks ago
- [ICML 2024] CLLMs: Consistency Large Language Models☆390Updated 4 months ago
- Triton-based implementation of Sparse Mixture of Experts.☆211Updated 4 months ago
- Microsoft Automatic Mixed Precision Library☆590Updated 6 months ago
- This repository contains the experimental PyTorch native float8 training UX☆222Updated 8 months ago
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆712Updated 6 months ago
- Some preliminary explorations of Mamba's context scaling.☆212Updated last year
- ☆413Updated this week
- PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention…☆288Updated 11 months ago
- Efficient LLM Inference over Long Sequences☆366Updated 2 months ago
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆229Updated last month
- 🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"☆621Updated 3 weeks ago
- Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793☆404Updated this week
- Explorations into some recent techniques surrounding speculative decoding☆253Updated 3 months ago
- USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference☆470Updated last week
- Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers☆207Updated 7 months ago
- Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAI☆278Updated 3 weeks ago
- Annotated version of the Mamba paper☆480Updated last year
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆230Updated 2 months ago
- Pipeline Parallelism for PyTorch☆762Updated 7 months ago
- Triton implementation of FlashAttention2 that adds Custom Masks.☆107Updated 8 months ago
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆313Updated 4 months ago
- Repo for "Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture"☆549Updated 3 months ago
- ☆182Updated this week