LIONS-EPFL / LIONLinks
Linear Attention for Efficient Bidirectional Sequence Modeling
☆15Updated 8 months ago
Alternatives and similar repositories for LION
Users that are interested in LION are comparing it to the libraries listed below
Sorting:
- HGRN2: Gated Linear RNNs with State Expansion☆56Updated last year
- Code for "Theoretical Foundations of Deep Selective State-Space Models" (NeurIPS 2024)☆15Updated last year
- Code for the paper "Cottention: Linear Transformers With Cosine Attention"☆20Updated 2 months ago
- [ICML-2025] We introduce Lie group Relative position Encodings (LieRE) that goes beyond RoPE in supporting n-dimensional inputs.☆14Updated 5 months ago
- Unofficial implementation of paper : Exploring the Space of Key-Value-Query Models with Intention☆12Updated 2 years ago
- A Triton Kernel for incorporating Bi-Directionality in Mamba2☆77Updated last year
- ☆62Updated last year
- [ICLR 2025] Official Code Release for Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation☆48Updated 10 months ago
- ☆32Updated 2 years ago
- Scalable and Stable Parallelization of Nonlinear RNNS☆28Updated 3 months ago
- ☆35Updated last year
- Unofficial Implementation of Selective Attention Transformer☆20Updated last year
- Official implementation of "Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers"☆170Updated last year
- [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…☆66Updated last year
- Efficient PScan implementation in PyTorch☆17Updated 2 years ago
- Stick-breaking attention☆62Updated 6 months ago
- ☆33Updated last year
- Open-sourcing code associated with the AAAI-25 paper "On the Expressiveness and Length Generalization of Selective State-Space Models on …☆15Updated 4 months ago
- ☆24Updated last year
- Here we will test various linear attention designs.☆62Updated last year
- ☆17Updated 7 months ago
- User-friendly implementation of the Mixture-of-Sparse-Attention (MoSA). MoSA selects distinct tokens for each head with expert choice rou…☆28Updated 8 months ago
- Reference implementation of "Softmax Attention with Constant Cost per Token" (Heinsen, 2024)☆24Updated last year
- Official code for the paper "Attention as a Hypernetwork"☆47Updated last year
- Why Do We Need Weight Decay in Modern Deep Learning? [NeurIPS 2024]☆70Updated last year
- Parallel Associative Scan for Language Models☆18Updated 2 years ago
- Experiments on the impact of depth in transformers and SSMs.☆40Updated 3 months ago
- Flash-Linear-Attention models beyond language☆21Updated 5 months ago
- Official Code Repository for the paper "Key-value memory in the brain"☆31Updated 11 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆27Updated last year