Alternatives and similar repositories for RWKV-howto:

Users that are interested in RWKV-howto are comparing it to the libraries listed below

• RobertCsordas / moe
  Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"
  ☆36Updated last year
• proger / hippogriff
  Griffin MQA + Hawk Linear RNN Hybrid
  ☆85Updated 9 months ago
• BlinkDL / LinearAttentionArena
  Here we will test various linear attention designs.
  ☆58Updated 9 months ago
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆52Updated 5 months ago
• wdlctc / mini-s
  ☆43Updated 3 months ago
• codekansas / rwkv
  RWKV model implementation
  ☆37Updated last year
• kyegomez / SimpleMamba
  Implementation of a modular, high-performance, and simplistic mamba for high-speed applications
  ☆33Updated 2 months ago
• recursal / GoldFinch-paper
  GoldFinch and other hybrid transformer components
  ☆43Updated 6 months ago
• OpenNLPLab / HGRN
  [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…
  ☆62Updated 9 months ago
• RobertCsordas / moe_layer
  sigma-MoE layer
  ☆18Updated last year
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆113Updated 3 months ago
• eth-easl / fmengine
  Utilities for Training Very Large Models
  ☆57Updated 4 months ago
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆96Updated 4 months ago
• tobiaskatsch / GatedLinearRNN
  ☆26Updated 11 months ago
• sustcsonglin / gated_linear_attention_layer
  ☆32Updated last year
• RWKV / RWKV-LM
  RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best…
  ☆21Updated 9 months ago
• lucidrains / gateloop-transformer
  Implementation of GateLoop Transformer in Pytorch and Jax
  ☆87Updated 7 months ago
• Oxen-AI / mamba-dive
  This is the code that went into our practical dive using mamba as information extraction
  ☆51Updated last year
• Itamarzimm / UnifiedImplicitAttnRepr
  ☆36Updated 8 months ago
• sustcsonglin / mamba-triton
  ☆46Updated last year
• ScalingIntelligence / large_language_monkeys
  ☆80Updated 4 months ago
• lucidrains / mirasol-pytorch
  Implementation of 🌻 Mirasol, SOTA Multimodal Autoregressive model out of Google Deepmind, in Pytorch
  ☆88Updated last year
• nanowell / Q-Sparse-LLM
  My Implementation of Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
  ☆31Updated 5 months ago
• xiayuqing0622 / flex_head_fa
  Fast and memory-efficient exact attention
  ☆57Updated last month
• geronimi73 / mamba
  ☆32Updated last year
• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆51Updated this week
• EleutherAI / mdl
  Minimum Description Length probing for neural network representations
  ☆18Updated this week
• alxndrTL / othello_mamba
  Evaluating the Mamba architecture on the Othello game
  ☆44Updated 9 months ago
• dangxingyu / rnn-icrag
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆25Updated 9 months ago
• lucidrains / pause-transformer
  Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount…
  ☆52Updated last year