Alternatives and similar repositories for RetNet

Users that are interested in RetNet are comparing it to the libraries listed below

• fkodom / yet-another-retnet
  A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (http…
  ☆106Updated last year
• kyegomez / MoE-Mamba
  Implementation of MoE Mamba from the paper: "MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts" in Pytorch and Ze…
  ☆113Updated last week
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆55Updated last year
• kyegomez / MambaTransformer
  Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling
  ☆207Updated last week
• WailordHe / DenseSSM
  A repository for DenseSSMs
  ☆89Updated last year
• kyegomez / Jamba
  PyTorch Implementation of Jamba: "Jamba: A Hybrid Transformer-Mamba Language Model"
  ☆192Updated last week
• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆56Updated last week
• piotrpiekos / MoSA
  User-friendly implementation of the Mixture-of-Sparse-Attention (MoSA). MoSA selects distinct tokens for each head with expert choice rou…
  ☆27Updated 6 months ago
• lucidrains / agent-attention-pytorch
  Implementation of Agent Attention in Pytorch
  ☆91Updated last year
• Montinger / Transformer-Workbench
  Playground for Transformers
  ☆53Updated last year
• Itamarzimm / UnifiedImplicitAttnRepr
  [ICLR 2025] Official Code Release for Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation
  ☆47Updated 8 months ago
• Weixin-Liang / Mixture-of-Mamba
  ☆50Updated 9 months ago
• badripatro / mamba360
  State Space Models
  ☆70Updated last year
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆119Updated last year
• kyegomez / MambaByte
  Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta
  ☆123Updated last week
• Arenaa / Accelerated-Generation-Techniques
  This repository contains papers for a comprehensive survey on accelerated generation techniques in Large Language Models (LLMs).
  ☆11Updated last year
• kyegomez / SimpleMamba
  Implementation of a modular, high-performance, and simplistic mamba for high-speed applications
  ☆36Updated 11 months ago
• kyegomez / MHMoE
  Community Implementation of the paper: "Multi-Head Mixture-of-Experts" In PyTorch
  ☆26Updated last week
• prateekstark / retnet
  ☆14Updated 2 years ago
• vulus98 / Rethinking-attention
  My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing o…
  ☆44Updated 10 months ago
• kyegomez / xLSTM
  Implementation of xLSTM in Pytorch from the paper: "xLSTM: Extended Long Short-Term Memory"
  ☆118Updated last week
• akaashdash / kansformers
  ☆137Updated last year
• radarFudan / mamba
  ☆18Updated last year
• ambisinister / mla-experiments
  Experiments on Multi-Head Latent Attention
  ☆98Updated last year
• AwesomeSeq / Comba-triton
  ☆49Updated 4 months ago
• jlamprou / Infini-Attention
  Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M cont…
  ☆84Updated last year
• tanaymeh / mamba-train
  A single repo with all scripts and utils to train / fine-tune the Mamba model with or without FIM
  ☆59Updated last year
• krafton-ai / mambaformer-icl
  MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248
  ☆57Updated last year
• lhallee / Multi_Head_Mixture_of_Experts__MH-MOE
  ☆29Updated last year
• fkodom / soft-mixture-of-experts
  PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
  ☆78Updated 2 years ago