lucidrains / routing-transformer
Fully featured implementation of Routing Transformer
☆284Updated 3 years ago
Related projects ⓘ
Alternatives and complementary repositories for routing-transformer
- Sinkhorn Transformer - Practical implementation of Sparse Sinkhorn Attention☆253Updated 3 years ago
- My take on a practical implementation of Linformer for Pytorch.☆407Updated 2 years ago
- Official PyTorch Implementation of Long-Short Transformer (NeurIPS 2021).☆222Updated 2 years ago
- ☆212Updated 4 years ago
- Implementation of Linformer for Pytorch☆257Updated 10 months ago
- DeLighT: Very Deep and Light-Weight Transformers☆467Updated 4 years ago
- Understanding the Difficulty of Training Transformers☆328Updated 2 years ago
- ☆365Updated last year
- Pytorch implementation of Compressive Transformers, from Deepmind☆157Updated 3 years ago
- Code for Multi-Head Attention: Collaborate Instead of Concatenate☆150Updated last year
- Transformer based on a variant of attention that is linear complexity in respect to sequence length☆698Updated 6 months ago
- Unofficial implementation of Google's FNet: Mixing Tokens with Fourier Transforms☆251Updated 3 years ago
- An implementation of local windowed attention for language modeling☆384Updated 2 months ago
- Implementation of https://arxiv.org/abs/1904.00962☆369Updated 3 years ago
- An implementation of Performer, a linear attention-based transformer, in Pytorch☆1,098Updated 2 years ago
- Implementation of Long-Short Transformer, combining local and global inductive biases for attention over long sequences, in Pytorch☆116Updated 3 years ago
- The entmax mapping and its loss, a family of sparse softmax alternatives.☆416Updated 4 months ago
- Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning☆155Updated 9 months ago
- Implementation of Feedback Transformer in Pytorch☆104Updated 3 years ago
- Implementation of Memformer, a Memory-augmented Transformer, in Pytorch☆106Updated 4 years ago
- Long Range Arena for Benchmarking Efficient Transformers☆729Updated 11 months ago
- This is a repository with the code for the ACL 2019 paper "Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, t…☆302Updated 3 years ago
- A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch☆222Updated last year
- Official codebase for Pretrained Transformers as Universal Computation Engines.☆245Updated 2 years ago
- Transformers without Tears: Improving the Normalization of Self-Attention☆130Updated 5 months ago
- Is the attention layer even necessary? (https://arxiv.org/abs/2105.02723)☆480Updated 3 years ago
- Trains Transformer model variants. Data isn't shuffled between batches.☆141Updated 2 years ago
- [ICML 2020] code for "PowerNorm: Rethinking Batch Normalization in Transformers" https://arxiv.org/abs/2003.07845☆119Updated 3 years ago
- PyTorch Implementation of OpenAI's Image GPT☆254Updated last year
- Sequence modeling with Mega.☆298Updated last year