Alternatives and similar repositories for RetNet

Users that are interested in RetNet are comparing it to the libraries listed below

• invictus717 / MetaTransformer
  Meta-Transformer for Unified Multimodal Learning
  ☆1,650Updated 2 years ago
• microsoft / torchscale
  Foundation Architecture for (M)LLMs
  ☆3,133Updated last year
• alxndrTL / mamba.py
  A simple and efficient Mamba implementation in pure PyTorch and MLX.
  ☆1,411Updated last year
• syncdoth / RetNet
  Huggingface compatible implementation of RetNet (Retentive Networks, https://arxiv.org/pdf/2307.08621.pdf) including parallel, recurrent,…
  ☆227Updated last year
• johnma2006 / mamba-minimal
  Simple, minimal implementation of the Mamba SSM in one file of PyTorch.
  ☆2,912Updated last year
• Liuhong99 / Sophia
  The official implementation of “Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”
  ☆981Updated last year
• test-time-training / ttt-lm-pytorch
  Official PyTorch implementation of Learning to (Learn at Test Time): RNNs with Expressive Hidden States
  ☆1,311Updated last year
• lucidrains / MEGABYTE-pytorch
  Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch
  ☆654Updated last year
• lucidrains / rotary-embedding-torch
  Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch
  ☆794Updated 6 months ago
• lucidrains / lion-pytorch
  🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch
  ☆2,182Updated last year
• kyegomez / zeta
  Build high-performance AI models with modular building blocks
  ☆577Updated last week
• kyegomez / LongNet
  Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"
  ☆714Updated 2 years ago
• state-spaces / s4
  Structured state space sequence models
  ☆2,827Updated last year
• bobby-he / simplified_transformers
  ☆293Updated last year
• lucidrains / mixture-of-experts
  A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models
  ☆844Updated 2 years ago
• radarFudan / Awesome-state-space-models
  Collection of papers on state-space models
  ☆615Updated 2 months ago
• redotvideo / mamba-chat
  Mamba-Chat: A chat LLM based on the state-space model architecture 🐍
  ☆938Updated last year
• davidmrau / mixture-of-experts
  PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538
  ☆1,222Updated last year
• HazyResearch / safari
  Convolutions for Sequence Modeling
  ☆909Updated last year
• Ma-Lab-Berkeley / CRATE
  Code for CRATE (Coding RAte reduction TransformEr).
  ☆1,266Updated last year
• princeton-nlp / MeZO
  [NeurIPS 2023] MeZO: Fine-Tuning Language Models with Just Forward Passes. https://arxiv.org/abs/2305.17333
  ☆1,140Updated 2 years ago
• sail-sg / Adan
  Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
  ☆806Updated 7 months ago
• OFA-Sys / ONE-PEACE
  A general representation model across vision, audio, language modalities. Paper: ONE-PEACE: Exploring One General Representation Model To…
  ☆1,064Updated last year
• yyyujintang / Awesome-Mamba-Papers
  Awesome Papers related to Mamba.
  ☆1,389Updated last year
• GistNoesis / FourierKAN
  ☆747Updated last year
• OpenLMLab / LOMO
  LOMO: LOw-Memory Optimization
  ☆988Updated last year
• jiachenzhu / DyT
  Code release for DynamicTanh (DyT)
  ☆1,034Updated 9 months ago
• hustvl / Vim
  [ICML 2024] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model
  ☆3,780Updated 11 months ago
• Dao-AILab / causal-conv1d
  Causal depthwise conv1d in CUDA, with a PyTorch interface
  ☆707Updated 2 weeks ago
• lucidrains / linear-attention-transformer
  Transformer based on a variant of attention that is linear complexity in respect to sequence length
  ☆825Updated last year