DonRL10 / RetNet
an implementation of paper"Retentive Network: A Successor to Transformer for Large Language Models" https://arxiv.org/pdf/2307.08621.pdf
☆12Updated last year
Alternatives and similar repositories for RetNet:
Users that are interested in RetNet are comparing it to the libraries listed below
- [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…☆64Updated 11 months ago
- ☆52Updated 8 months ago
- ☆23Updated 6 months ago
- A large-scale RWKV v6, v7(World, ARWKV, PRWKV) inference. Capable of inference by combining multiple states(Pseudo MoE). Easy to deploy o…☆33Updated last week
- ☆27Updated 8 months ago
- HGRN2: Gated Linear RNNs with State Expansion☆54Updated 7 months ago
- ☆32Updated last year
- Official Code Repository for the paper "Key-value memory in the brain"☆24Updated last month
- GoldFinch and other hybrid transformer components☆45Updated 8 months ago
- Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"☆36Updated last year
- Scaling Sparse Fine-Tuning to Large Language Models☆16Updated last year
- Implementation of GateLoop Transformer in Pytorch and Jax☆87Updated 9 months ago
- PyTorch implementation of Retentive Network: A Successor to Transformer for Large Language Models☆15Updated last year
- ☆64Updated 7 months ago
- Randomized Positional Encodings Boost Length Generalization of Transformers☆80Updated last year
- Engineering the state of RNN language models (Mamba, RWKV, etc.)☆32Updated 10 months ago
- Efficient PScan implementation in PyTorch☆16Updated last year
- My Implementation of Q-Sparse: All Large Language Models can be Fully Sparsely-Activated☆31Updated 7 months ago
- Contextual Position Encoding but with some custom CUDA Kernels https://arxiv.org/abs/2405.18719☆22Updated 9 months ago
- ☆12Updated 11 months ago
- Griffin MQA + Hawk Linear RNN Hybrid☆85Updated 11 months ago
- Tiled Flash Linear Attention library for fast and efficient mLSTM Kernels.☆50Updated last week
- Triton Implementation of HyperAttention Algorithm☆47Updated last year
- Combining SOAP and MUON☆13Updated last month
- Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"☆52Updated 2 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆26Updated 11 months ago
- ☆47Updated last year
- ☆13Updated last year
- ☆19Updated 2 years ago
- The open source implementation of "Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers"☆20Updated last year