shreyansh26 / Attention-Mask-PatternsLinks
Using FlexAttention to compute attention with different masking patterns
☆44Updated 10 months ago
Alternatives and similar repositories for Attention-Mask-Patterns
Users that are interested in Attention-Mask-Patterns are comparing it to the libraries listed below
Sorting:
- Triton Implementation of HyperAttention Algorithm☆48Updated last year
- ☆56Updated last year
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆77Updated 9 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆61Updated 9 months ago
- Simple and efficient pytorch-native transformer training and inference (batched)☆78Updated last year
- ☆82Updated 11 months ago
- The official repository for SkyLadder: Better and Faster Pretraining via Context Window Scheduling☆33Updated last week
- ☆38Updated 3 months ago
- A repository for research on medium sized language models.☆78Updated last year
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆27Updated last year
- The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.☆82Updated 2 weeks ago
- Here we will test various linear attention designs.☆62Updated last year
- The repository contains code for Adaptive Data Optimization☆25Updated 7 months ago
- The evaluation framework for training-free sparse attention in LLMs☆86Updated last month
- ☆20Updated last year
- DPO, but faster 🚀☆43Updated 7 months ago
- Language models scale reliably with over-training and on downstream tasks☆97Updated last year
- ☆49Updated last year
- ☆51Updated last month
- Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…☆56Updated this week
- ☆51Updated last year
- Code for the paper "On the Expressivity Role of LayerNorm in Transformers' Attention" (Findings of ACL'2023)☆56Updated 10 months ago
- ☆37Updated last year
- JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)☆35Updated last year
- GoldFinch and other hybrid transformer components☆46Updated last year
- CUDA and Triton implementations of Flash Attention with SoftmaxN.☆71Updated last year
- Official implementation of the ICML 2024 paper RoSA (Robust Adaptation)☆41Updated last year
- ☆33Updated last year
- Code for the arXiv preprint "The Unreasonable Effectiveness of Easy Training Data"☆48Updated last year
- ☆81Updated last year