Alternatives and similar repositories for RetNet:

Users that are interested in RetNet are comparing it to the libraries listed below

• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆52Updated 5 months ago
• kyegomez / MoE-Mamba
  Implementation of MoE Mamba from the paper: "MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts" in Pytorch and Ze…
  ☆92Updated this week
• kyegomez / MambaFormer
  Implementation of MambaFormer in Pytorch ++ Zeta from the paper: "Can Mamba Learn How to Learn? A Comparative Study on In-Context Learnin…
  ☆20Updated this week
• DonRL10 / RetNet
  an implementation of paper"Retentive Network: A Successor to Transformer for Large Language Models" https://arxiv.org/pdf/2307.08621.pdf
  ☆12Updated last year
• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆51Updated this week
• kyegomez / SimpleMamba
  Implementation of a modular, high-performance, and simplistic mamba for high-speed applications
  ☆33Updated 2 months ago
• knotgrass / attention
  several types of attention modules written in PyTorch for learning purposes
  ☆44Updated 3 months ago
• fkodom / yet-another-retnet
  A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (http…
  ☆104Updated last year
• vulus98 / Rethinking-attention
  My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing o…
  ☆44Updated last month
• kyegomez / MHMoE
  Community Implementation of the paper: "Multi-Head Mixture-of-Experts" In PyTorch
  ☆20Updated this week
• Itamarzimm / UnifiedImplicitAttnRepr
  ☆36Updated 8 months ago
• kyegomez / MultiQueryAttention
  This is a simple torch implementation of the high performance Multi-Query Attention
  ☆16Updated last year
• prateekstark / retnet
  ☆14Updated last year
• uclaml / MoE
  Towards Understanding the Mixture-of-Experts Layer in Deep Learning
  ☆22Updated last year
• WailordHe / DenseSSM
  A repository for DenseSSMs
  ☆86Updated 9 months ago
• radarFudan / mamba
  ☆15Updated 3 months ago
• assafbk / DeciMamba
  Official Implementation Of The Paper: `DeciMamba: Exploring the Length Extrapolation Potential of Mamba'
  ☆22Updated 6 months ago
• kyegomez / TTL
  Pytorch Implementation of the paper: "Learning to (Learn at Test Time): RNNs with Expressive Hidden States"
  ☆24Updated this week
• OpenMOSE / RWKV-Infer
  A large-scale RWKV v6, v7 inference. Capable of inference by combining multiple states(Pseudo MoE). Easy to deploy on docker. Supports tr…
  ☆25Updated last week
• kyegomez / MambaByte
  Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta
  ☆114Updated this week
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆96Updated 4 months ago
• quiqi / relu_kan
  ☆80Updated 7 months ago
• badripatro / mamba360
  State Space Models
  ☆64Updated 8 months ago
• lhallee / Multi_Head_Mixture_of_Experts__MH-MOE
  ☆21Updated 3 months ago
• apapiu / mamba_small_bench
  Trying out the Mamba architecture on small examples (cifar-10, shakespeare char level etc.)
  ☆43Updated last year
• kyegomez / xLSTM
  Implementation of xLSTM in Pytorch from the paper: "xLSTM: Extended Long Short-Term Memory"
  ☆115Updated this week
• lucidrains / agent-attention-pytorch
  Implementation of Agent Attention in Pytorch
  ☆89Updated 6 months ago
• myscience / mamba
  Pytorch (Lightning) implementation of the Mamba model
  ☆23Updated 9 months ago
• Doraemonzzz / hgru2-pytorch
  ☆24Updated 4 months ago
• Montinger / Transformer-Workbench
  Playground for Transformers
  ☆47Updated last year