Hannibal046 / RWKV-howto
possibly useful materials for learning RWKV language model.
☆25Updated last year
Related projects ⓘ
Alternatives and complementary repositories for RWKV-howto
- Here we will test various linear attention designs.☆56Updated 6 months ago
- Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"☆36Updated last year
- RWKV model implementation☆38Updated last year
- ☆46Updated last month
- Triton Implementation of HyperAttention Algorithm☆46Updated 11 months ago
- Griffin MQA + Hawk Linear RNN Hybrid☆85Updated 6 months ago
- Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts☆108Updated last month
- ☆33Updated 5 months ago
- NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference☆61Updated last month
- Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"☆50Updated last week
- CUDA implementation of autoregressive linear attention, with all the latest research findings☆43Updated last year
- ☆31Updated 10 months ago
- RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best…☆21Updated 7 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆24Updated 7 months ago
- ☆62Updated 3 months ago
- Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount…☆49Updated last year
- Implementation of Infini-Transformer in Pytorch☆104Updated last month
- ☆14Updated 11 months ago
- HGRN2: Gated Linear RNNs with State Expansion☆49Updated 3 months ago
- [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…☆61Updated 6 months ago
- Mixture of A Million Experts☆32Updated 3 months ago
- Simple and efficient pytorch-native transformer training and inference (batched)☆61Updated 7 months ago
- ☆49Updated last year
- Official Implementation Of The Paper: `DeciMamba: Exploring the Length Extrapolation Potential of Mamba'☆20Updated 3 months ago
- Linear Attention Sequence Parallelism (LASP)☆64Updated 5 months ago
- Stick-breaking attention☆34Updated last week
- Implementation of GateLoop Transformer in Pytorch and Jax☆86Updated 5 months ago
- A simple implementation of [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752)☆20Updated 10 months ago