lena-voita / the-story-of-heads
This is a repository with the code for the ACL 2019 paper "Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned" and the ACL 2021 paper "Analyzing Source and Target Contributions to NMT Predictions".
☆296Updated 3 years ago
Related projects: ⓘ
- Code for the paper "Are Sixteen Heads Really Better than One?"☆165Updated 4 years ago
- Understanding the Difficulty of Training Transformers☆325Updated 2 years ago
- Repository containing code for "How to Train BERT with an Academic Budget" paper☆309Updated last year
- For the code release of our arXiv paper "Revisiting Few-sample BERT Fine-tuning" (https://arxiv.org/abs/2006.05987).☆184Updated last year
- ☆476Updated 10 months ago
- [ACL 2022] Structured Pruning Learns Compact and Accurate Models https://arxiv.org/abs/2204.00408☆188Updated last year
- Pretrain and finetune ELECTRA with fastai and huggingface. (Results of the paper replicated !)☆323Updated 8 months ago
- ☆311Updated 3 years ago
- ☆292Updated last year
- Tracking the progress in non-autoregressive generation (translation, transcription, etc.)☆305Updated last year
- ☆450Updated 3 years ago
- A masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a…☆238Updated 3 years ago
- Fully featured implementation of Routing Transformer☆282Updated 2 years ago
- A PyTorch implementation of Transformer in "Attention is All You Need"☆103Updated 3 years ago
- Transformer with Untied Positional Encoding (TUPE). Code of paper "Rethinking Positional Encoding in Language Pre-training". Improve exis…☆249Updated 2 years ago
- An original implementation of "MetaICL Learning to Learn In Context" by Sewon Min, Mike Lewis, Luke Zettlemoyer and Hannaneh Hajishirzi☆249Updated last year
- ☆308Updated 2 years ago
- PyTorch implementation of BERT in "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"☆94Updated 5 years ago
- [ACL 2021] LM-BFF: Better Few-shot Fine-tuning of Language Models https://arxiv.org/abs/2012.15723☆714Updated 2 years ago
- Interpretable Evaluation for AI Systems☆359Updated last year
- Code for the NAACL 2022 long paper "DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings"☆290Updated last year
- Code for Multi-Head Attention: Collaborate Instead of Concatenate☆148Updated last year
- Tools for checking ACL paper submissions☆554Updated 4 months ago
- ⛵️The official PyTorch implementation for "BERT-of-Theseus: Compressing BERT by Progressive Module Replacing" (EMNLP 2020).☆310Updated last year
- Neural Text Generation with Unlikelihood Training☆311Updated 3 years ago
- Transformers without Tears: Improving the Normalization of Self-Attention☆128Updated 3 months ago
- DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference☆151Updated 2 years ago
- Run Effective Large Batch Contrastive Learning Beyond GPU/TPU Memory Constraint☆342Updated 5 months ago
- Optimus: the first large-scale pre-trained VAE language model☆367Updated last year
- A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch☆222Updated last year