shawntan / SUT
Repository for Sparse Universal Transformers
☆17Updated last year
Related projects ⓘ
Alternatives and complementary repositories for SUT
- ☆44Updated last year
- Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"☆36Updated last year
- Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount…☆49Updated last year
- The Energy Transformer block, in JAX☆53Updated 11 months ago
- ☆21Updated 2 months ago
- ☆29Updated 7 months ago
- ☆50Updated 6 months ago
- Official code for the paper "Attention as a Hypernetwork"☆23Updated 5 months ago
- ☆46Updated last month
- ☆45Updated 9 months ago
- ☆29Updated this week
- ☆31Updated 10 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆24Updated 7 months ago
- Stick-breaking attention☆34Updated 2 weeks ago
- Source code for the paper "Positional Attention: Out-of-Distribution Generalization and Expressivity for Neural Algorithmic Reasoning"☆14Updated this week
- Unofficial but Efficient Implementation of "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" in JAX☆79Updated 10 months ago
- ☆11Updated last year
- ☆24Updated 8 months ago
- ☆26Updated last month
- HGRN2: Gated Linear RNNs with State Expansion☆49Updated 3 months ago
- ☆25Updated last month
- A MAD laboratory to improve AI architecture designs 🧪☆95Updated 6 months ago
- An annotated implementation of the Hyena Hierarchy paper☆31Updated last year
- Implementation of Hyena Hierarchy in JAX☆10Updated last year
- ☆21Updated 2 years ago
- sigma-MoE layer☆18Updated 10 months ago
- PyTorch implementation for "Long Horizon Temperature Scaling", ICML 2023☆19Updated last year
- Triton Implementation of HyperAttention Algorithm☆46Updated 11 months ago
- Parallelizing non-linear sequential models over the sequence length☆45Updated 3 weeks ago
- ☆53Updated 10 months ago