MLGroupJLU / RWKV-SurveyLinks
The official GitHub page for the survey paper "A Survey of RWKV".
☆27Updated 5 months ago
Alternatives and similar repositories for RWKV-Survey
Users that are interested in RWKV-Survey are comparing it to the libraries listed below
Sorting:
- HGRN2: Gated Linear RNNs with State Expansion☆55Updated 10 months ago
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)☆28Updated 2 months ago
- [ICML 2024] When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models☆31Updated last year
- Triton implement of bi-directional (non-causal) linear attention☆50Updated 4 months ago
- Official implementation of RMoE (Layerwise Recurrent Router for Mixture-of-Experts)☆22Updated 10 months ago
- Pytorch Implementation of the paper: "Learning to (Learn at Test Time): RNNs with Expressive Hidden States"☆25Updated this week
- A repository for DenseSSMs☆87Updated last year
- ☆63Updated 4 months ago
- User-friendly implementation of the Mixture-of-Sparse-Attention (MoSA). MoSA selects distinct tokens for each head with expert choice rou…☆21Updated last month
- Offical implementation of "MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map" (NeurIPS2024 Oral)☆25Updated 5 months ago
- The open-source materials for paper "Sparsing Law: Towards Large Language Models with Greater Activation Sparsity".☆23Updated 7 months ago
- ☆21Updated 2 years ago
- [ICML 2025] Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization☆72Updated 3 weeks ago
- [ICML 2023] "Data Efficient Neural Scaling Law via Model Reusing" by Peihao Wang, Rameswar Panda, Zhangyang Wang☆14Updated last year
- This is a simple torch implementation of the high performance Multi-Query Attention☆17Updated last year
- My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing o…☆43Updated 6 months ago
- Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"☆55Updated 2 months ago
- ☆23Updated 9 months ago
- [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…☆65Updated last year
- [ICLR 2025] Official Code Release for Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation☆42Updated 3 months ago
- Official Code Repository for the paper "Key-value memory in the brain"☆26Updated 4 months ago
- ☆18Updated last week
- Official code for the paper "Attention as a Hypernetwork"☆39Updated last year
- Contextual Position Encoding but with some custom CUDA Kernels https://arxiv.org/abs/2405.18719☆22Updated last year
- ☆47Updated last year
- Inference Speed Benchmark for Learning to (Learn at Test Time): RNNs with Expressive Hidden States☆67Updated 11 months ago
- ☆16Updated 6 months ago
- Mixture of Attention Heads☆47Updated 2 years ago
- Official PyTorch implementation of "LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging" (ICML 2024)☆29Updated 10 months ago
- ☆43Updated 5 months ago