howard-hou / RWKV-XLinks
RWKV-X is a Linear Complexity Hybrid Language Model based on the RWKV architecture, integrating Sparse Attention to improve the model's long sequence processing capabilities.
☆46Updated 3 weeks ago
Alternatives and similar repositories for RWKV-X
Users that are interested in RWKV-X are comparing it to the libraries listed below
Sorting:
- Here we will test various linear attention designs.☆62Updated last year
- A large-scale RWKV v6, v7(World, PRWKV, Hybrid-RWKV) inference. Capable of inference by combining multiple states(Pseudo MoE). Easy to de…☆40Updated last week
- https://x.com/BlinkDL_AI/status/1884768989743882276☆28Updated 3 months ago
- RADLADS training code☆26Updated 3 months ago
- [ICLR 2025] Official PyTorch implementation of "Forgetting Transformer: Softmax Attention with a Forget Gate"☆118Updated last month
- ☆24Updated last week
- ☆83Updated 11 months ago
- My Implementation of Q-Sparse: All Large Language Models can be Fully Sparsely-Activated☆33Updated 11 months ago
- The evaluation framework for training-free sparse attention in LLMs☆86Updated last month
- DPO, but faster 🚀☆44Updated 8 months ago
- Repository for the Q-Filters method (https://arxiv.org/pdf/2503.02812)☆34Updated 5 months ago
- From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…☆47Updated 3 months ago
- Stick-breaking attention☆59Updated last month
- The official implementation for Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free☆46Updated 2 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆27Updated last year
- A repository for research on medium sized language models.☆78Updated last year
- ☆79Updated 5 months ago
- Reference implementation of "Softmax Attention with Constant Cost per Token" (Heinsen, 2024)☆24Updated last year
- [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…☆66Updated last year
- RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best…☆51Updated 4 months ago
- Awesome Triton Resources☆32Updated 3 months ago
- GoldFinch and other hybrid transformer components☆46Updated last year
- Triton implement of bi-directional (non-causal) linear attention☆52Updated 6 months ago
- ☆39Updated 4 months ago
- ☆49Updated last year
- ☆56Updated last year
- HGRN2: Gated Linear RNNs with State Expansion☆55Updated 11 months ago
- Tiny re-implementation of MDM in style of LLaDA and nano-gpt speedrun☆55Updated 4 months ago
- Flash-Linear-Attention models beyond language☆16Updated 2 weeks ago
- ☆60Updated 4 months ago