google-deepmind / randomized_positional_encodingsLinks
Randomized Positional Encodings Boost Length Generalization of Transformers
☆82Updated last year
Alternatives and similar repositories for randomized_positional_encodings
Users that are interested in randomized_positional_encodings are comparing it to the libraries listed below
Sorting:
- [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…☆64Updated last year
- ☆46Updated last year
- Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"☆37Updated last year
- Code for the paper "The Impact of Positional Encoding on Length Generalization in Transformers", NeurIPS 2023☆134Updated last year
- Griffin MQA + Hawk Linear RNN Hybrid☆86Updated last year
- some common Huggingface transformers in maximal update parametrization (µP)☆80Updated 3 years ago
- ☆64Updated 9 months ago
- Language models scale reliably with over-training and on downstream tasks☆97Updated last year
- ☆52Updated last year
- A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.☆67Updated 10 months ago
- ☆37Updated last year
- ☆54Updated 10 months ago
- Online Adaptation of Language Models with a Memory of Amortized Contexts (NeurIPS 2024)☆63Updated 10 months ago
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆97Updated 8 months ago
- ☆97Updated 2 years ago
- ☆144Updated 2 years ago
- A toolkit for scaling law research ⚖☆49Updated 4 months ago
- ☆81Updated last year
- Implementation of Infini-Transformer in Pytorch☆111Updated 5 months ago
- ☆18Updated 2 years ago
- Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount…☆53Updated last year
- ☆85Updated last year
- Code for paper "Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning"☆78Updated last year
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)☆155Updated last month
- Sparse Backpropagation for Mixture-of-Expert Training☆29Updated 11 months ago
- Official code release for "SuperBPE: Space Travel for Language Models"☆47Updated last week
- ☆92Updated 8 months ago
- Implementation of 🌻 Mirasol, SOTA Multimodal Autoregressive model out of Google Deepmind, in Pytorch☆88Updated last year
- Code for paper "Patch-Level Training for Large Language Models"☆86Updated 6 months ago
- ☆79Updated 9 months ago