tech-srl / layer_norm_expressivity_role
Code for the paper "On the Expressivity Role of LayerNorm in Transformers' Attention" (Findings of ACL'2023)
☆43Updated last year
Related projects: ⓘ
- Using FlexAttention to compute attention with different masking patterns☆28Updated last week
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆24Updated 5 months ago
- ☆29Updated last year
- ☆48Updated 3 months ago
- The official repository for our paper "The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns …☆15Updated 10 months ago
- Implementation of TableFormer, Robust Transformer Modeling for Table-Text Encoding, in Pytorch☆35Updated 2 years ago
- Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing☆47Updated 2 years ago
- ☆41Updated 2 months ago
- ☆40Updated 2 years ago
- ☆18Updated 3 months ago
- ☆42Updated this week
- ☆21Updated this week
- Code for the PAPA paper☆27Updated last year
- Simple and efficient pytorch-native transformer training and inference (batched)☆53Updated 5 months ago
- A framework to study AI models in Reasoning, Alignment, and use of Memory (RAM).☆15Updated last week
- Code for "Merging Text Transformers from Different Initializations"☆18Updated last month
- ☆37Updated last year
- ☆20Updated last year
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆42Updated last year
- Experiments for "A Closer Look at In-Context Learning under Distribution Shifts"☆20Updated last year
- ☆47Updated 3 months ago
- Implementation of some personal helper functions for Einops, my most favorite tensor manipulation library ❤️☆52Updated last year
- [ICML 24 NGSM workshop] Associative Recurrent Memory Transformer implementation and scripts for training and evaluating☆26Updated last week
- codebase for the SIMAT dataset and evaluation☆38Updated 2 years ago
- ☆30Updated 8 months ago
- Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)☆77Updated last year
- Official code for the paper "Attention as a Hypernetwork"☆20Updated 2 months ago
- ☆22Updated 6 months ago
- Repository for Skill Set Optimization☆12Updated last month
- Implementation of the model: "Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models" in PyTorch☆27Updated this week