Alternatives and similar repositories for RWKV-howto:

Users that are interested in RWKV-howto are comparing it to the libraries listed below

• RobertCsordas / moe
  Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"
  ☆36Updated last year
• BlinkDL / LinearAttentionArena
  Here we will test various linear attention designs.
  ☆59Updated 10 months ago
• OpenNLPLab / LASP
  Linear Attention Sequence Parallelism (LASP)
  ☆79Updated 9 months ago
• HazyResearch / prefix-linear-attention
  ☆50Updated 7 months ago
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆53Updated 6 months ago
• amirzandieh / HyperAttention
  Triton Implementation of HyperAttention Algorithm
  ☆47Updated last year
• kyleliang919 / Online-Subspace-Descent
  This repo is based on https://github.com/jiaweizzhao/GaLore
  ☆25Updated 5 months ago
• ZongqianLi / 500xCompressor
  Repository for the paper: 500xCompressor: Generalized Prompt Compression for Large Language Models
  ☆27Updated 6 months ago
• assafbk / DeciMamba
  DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)
  ☆23Updated 7 months ago
• kyegomez / SimpleMamba
  Implementation of a modular, high-performance, and simplistic mamba for high-speed applications
  ☆33Updated 3 months ago
• prateeky2806 / ComPEFT
  ☆25Updated last year
• kyegomez / Blockwise-Parallel-Transformer
  32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.
  ☆46Updated last year
• aniquetahir / JORA
  JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)
  ☆32Updated 10 months ago
• xiayuqing0622 / flex_head_fa
  Fast and memory-efficient exact attention
  ☆64Updated this week
• wdlctc / mini-s
  ☆48Updated 4 months ago
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆96Updated 5 months ago
• Itamarzimm / UnifiedImplicitAttnRepr
  [ICLR 2025] Official Code Release for Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation
  ☆37Updated this week
• eth-easl / fmengine
  Utilities for Training Very Large Models
  ☆58Updated 5 months ago
• proger / hippogriff
  Griffin MQA + Hawk Linear RNN Hybrid
  ☆85Updated 10 months ago
• tridao / flash-attention-wheels
  ☆46Updated last year
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆116Updated 4 months ago
• tobiaskatsch / GatedLinearRNN
  ☆27Updated last year
• glassroom / heinsen_attention
  Reference implementation of "Softmax Attention with Constant Cost per Token" (Heinsen, 2024)
  ☆24Updated 8 months ago
• tding1 / Efficient-LLM-Survey
  The Efficiency Spectrum of LLM
  ☆52Updated last year
• sustcsonglin / gated_linear_attention_layer
  ☆33Updated last year
• fkodom / soft-mixture-of-experts
  PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
  ☆71Updated last year
• dtunai / Griffin-Jax
  Jax implementation of "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆13Updated 9 months ago
• OpenNLPLab / HGRN
  [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…
  ☆64Updated 10 months ago
• RobertCsordas / moe_layer
  sigma-MoE layer
  ☆18Updated last year
• codekansas / rwkv
  RWKV model implementation
  ☆37Updated last year