ag1988 / top_k_attentionLinks
The accompanying code for "Memory-efficient Transformers via Top-k Attention" (Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, Jonathan Berant. SustaiNLP 2021).
☆70Updated 4 years ago
Alternatives and similar repositories for top_k_attention
Users that are interested in top_k_attention are comparing it to the libraries listed below
Sorting:
- [EVA ICLR'23; LARA ICML'22] Efficient attention mechanisms via control variates, random features, and importance sampling☆86Updated 2 years ago
- [EMNLP 2022] Official implementation of Transnormer in our EMNLP 2022 paper - The Devil in Linear Transformer☆62Updated 2 years ago
- This package implements THOR: Transformer with Stochastic Experts.☆65Updated 3 years ago
- Code for the paper "Query-Key Normalization for Transformers"☆48Updated 4 years ago
- ☆106Updated last year
- Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"☆120Updated last year
- Parameter Efficient Transfer Learning with Diff Pruning☆74Updated 4 years ago
- ☆21Updated 2 years ago
- Sparse Backpropagation for Mixture-of-Expert Training☆30Updated last year
- [KDD'22] Learned Token Pruning for Transformers☆100Updated 2 years ago
- Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)☆81Updated 2 years ago
- Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method (NeurIPS 2021)☆63Updated 3 years ago
- [ICLR 2023] Official implementation of Transnormer in our ICLR 2023 paper - Toeplitz Neural Network for Sequence Modeling☆80Updated last year
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆55Updated 2 years ago
- [NeurIPS 2023] Make Your Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning☆32Updated 2 years ago
- Retrieval as Attention☆83Updated 2 years ago
- Code for "Training Neural Networks with Fixed Sparse Masks" (NeurIPS 2021).☆59Updated 3 years ago
- FairSeq repo with Apollo optimizer☆114Updated last year
- ☆14Updated 2 years ago
- ☆33Updated 4 years ago
- ☆29Updated 2 years ago
- ☆56Updated last year
- Xmixers: A collection of SOTA efficient token/channel mixers☆14Updated 3 weeks ago
- ☆130Updated 3 years ago
- 32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.☆49Updated 2 years ago
- Block Sparse movement pruning☆81Updated 4 years ago
- (ACL-IJCNLP 2021) Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models.☆21Updated 3 years ago
- ☆31Updated 2 years ago
- ☆51Updated 2 years ago
- Code for the ACL-2022 paper "StableMoE: Stable Routing Strategy for Mixture of Experts"☆50Updated 3 years ago