tech-srl / layer_norm_expressivity_roleLinks
Code for the paper "On the Expressivity Role of LayerNorm in Transformers' Attention" (Findings of ACL'2023)
☆55Updated 9 months ago
Alternatives and similar repositories for layer_norm_expressivity_role
Users that are interested in layer_norm_expressivity_role are comparing it to the libraries listed below
Sorting:
- Using FlexAttention to compute attention with different masking patterns☆44Updated 9 months ago
- A basic pure pytorch implementation of flash attention☆16Updated 8 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆27Updated last year
- ☆51Updated last year
- ☆29Updated 2 years ago
- Utilities for Training Very Large Models☆58Updated 9 months ago
- ☆37Updated 2 years ago
- Triton Implementation of HyperAttention Algorithm☆48Updated last year
- ☆23Updated 8 months ago
- Implementation of TableFormer, Robust Transformer Modeling for Table-Text Encoding, in Pytorch☆39Updated 3 years ago
- Experimental scripts for researching data adaptive learning rate scheduling.☆23Updated last year
- Here we will test various linear attention designs.☆59Updated last year
- Code for LaMPP: Language Models as Probabilistic Priors for Perception and Action☆37Updated 2 years ago
- Easily run PyTorch on multiple GPUs & machines☆46Updated 3 months ago
- ☆37Updated last year
- We study toy models of skill learning.☆28Updated 5 months ago
- LL3M: Large Language and Multi-Modal Model in Jax☆72Updated last year
- A repository for research on medium sized language models.☆76Updated last year
- ☆33Updated 5 months ago
- GoldFinch and other hybrid transformer components☆45Updated 11 months ago
- This repo is based on https://github.com/jiaweizzhao/GaLore☆28Updated 9 months ago
- code for paper "Accessing higher dimensions for unsupervised word translation"☆21Updated 2 years ago
- ☆55Updated 11 months ago
- ☆20Updated last year
- ☆32Updated last year
- Implementation of Gradient Agreement Filtering, from Chaubard et al. of Stanford, but for single machine microbatches, in Pytorch☆25Updated 5 months ago
- Implementation of Infini-Transformer in Pytorch☆111Updated 5 months ago
- Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing☆50Updated 3 years ago
- implementation of dualformer☆17Updated 3 months ago
- Implementation of a holodeck, written in Pytorch☆18Updated last year