Alternatives and similar repositories for attention:

Users that are interested in attention are comparing it to the libraries listed below

• ambisinister / mla-experiments
  Experiments on Multi-Head Latent Attention
  ☆87Updated 8 months ago
• lucidrains / infini-transformer-pytorch
  Implementation of Infini-Transformer in Pytorch
  ☆110Updated 3 months ago
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆54Updated 8 months ago
• tridao / flash-attention-wheels
  ☆48Updated last year
• nyunAI / Faster-LLM-Survey
  ☆41Updated last year
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆90Updated this week
• Montinger / Transformer-Workbench
  Playground for Transformers
  ☆49Updated last year
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆97Updated 6 months ago
• WailordHe / DenseSSM
  A repository for DenseSSMs
  ☆87Updated last year
• radarFudan / mamba
  ☆16Updated 6 months ago
• IST-DASLab / RoSA
  Official implementation of the ICML 2024 paper RoSA (Robust Adaptation)
  ☆40Updated last year
• krafton-ai / mambaformer-icl
  MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248
  ☆51Updated 10 months ago
• The-AI-Summer / pytorch-ddp
  code for the ddp tutorial
  ☆32Updated 3 years ago
• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆52Updated 3 weeks ago
• fkodom / grouped-query-attention-pytorch
  (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …
  ☆165Updated 11 months ago
• ziplab / QLLM
  [ICLR 2024] This is the official PyTorch implementation of "QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Mod…
  ☆26Updated last year
• jlamprou / Infini-Attention
  Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M cont…
  ☆82Updated 11 months ago
• kyegomez / MultiQueryAttention
  This is a simple torch implementation of the high performance Multi-Query Attention
  ☆16Updated last year
• yuzhenmao / IceFormer
  Implementation of IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs (ICLR 2024).
  ☆24Updated 10 months ago
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆118Updated 6 months ago
• Doraemonzzz / Awesome-Triton-Resources
  Awesome Triton Resources
  ☆26Updated 3 weeks ago
• amirzandieh / HyperAttention
  Triton Implementation of HyperAttention Algorithm
  ☆47Updated last year
• kyegomez / Infini-attention
  Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…
  ☆55Updated this week
• foundation-model-stack / bamba
  Train, tune, and infer Bamba model
  ☆88Updated this week
• lucidrains / tableformer-pytorch
  Implementation of TableFormer, Robust Transformer Modeling for Table-Text Encoding, in Pytorch
  ☆37Updated 3 years ago
• pprp / Pruner-Zero
  Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMs
  ☆81Updated 5 months ago
• YeonwooSung / Pytorch_mixture-of-experts
  PyTorch implementation of moe, which stands for mixture of experts
  ☆43Updated 4 years ago
• MLGroupJLU / RWKV-Survey
  The official GitHub page for the survey paper "A Survey of RWKV".
  ☆25Updated 3 months ago
• Nicolas-BZRD / llm-recipes
  ☆28Updated last year
• kyegomez / SparseAttention
  Pytorch Implementation of the sparse attention from the paper: "Generating Long Sequences with Sparse Transformers"
  ☆78Updated 2 weeks ago