Alternatives and similar repositories for attention

Users that are interested in attention are comparing it to the libraries listed below

• ambisinister / mla-experiments
  Experiments on Multi-Head Latent Attention
  ☆92Updated 10 months ago
• Montinger / Transformer-Workbench
  Playground for Transformers
  ☆51Updated last year
• fkodom / grouped-query-attention-pytorch
  (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …
  ☆170Updated last year
• fkodom / soft-mixture-of-experts
  PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
  ☆73Updated last year
• tridao / flash-attention-wheels
  ☆50Updated last year
• WailordHe / DenseSSM
  A repository for DenseSSMs
  ☆87Updated last year
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆98Updated 8 months ago
• YeonwooSung / Pytorch_mixture-of-experts
  PyTorch implementation of moe, which stands for mixture of experts
  ☆45Updated 4 years ago
• nyunAI / Faster-LLM-Survey
  ☆42Updated last year
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆95Updated 2 weeks ago
• kyegomez / SparseAttention
  Pytorch Implementation of the sparse attention from the paper: "Generating Long Sequences with Sparse Transformers"
  ☆84Updated 2 months ago
• Arenaa / Accelerated-Generation-Techniques
  This repository contains papers for a comprehensive survey on accelerated generation techniques in Large Language Models (LLMs).
  ☆11Updated last year
• radarFudan / mamba
  ☆18Updated 7 months ago
• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆55Updated 2 months ago
• test-time-training / ttt-lm-kernels
  Inference Speed Benchmark for Learning to (Learn at Test Time): RNNs with Expressive Hidden States
  ☆67Updated 11 months ago
• berlino / gated_linear_attention
  ☆105Updated last year
• lucidrains / infini-transformer-pytorch
  Implementation of Infini-Transformer in Pytorch
  ☆111Updated 5 months ago
• MLGroupJLU / RWKV-Survey
  The official GitHub page for the survey paper "A Survey of RWKV".
  ☆27Updated 5 months ago
• juvi21 / CoPE-cuda
  Contextual Position Encoding but with some custom CUDA Kernels https://arxiv.org/abs/2405.18719
  ☆22Updated last year
• mengxiayu / LLMSuperWeight
  Code for studying the super weight in LLM
  ☆107Updated 6 months ago
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆55Updated 10 months ago
• pprp / Pruner-Zero
  [ICML24] Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMs
  ☆83Updated 7 months ago
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆119Updated 8 months ago
• kyegomez / Blockwise-Parallel-Transformer
  32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.
  ☆48Updated 2 years ago
• OpenNLPLab / LASP
  Linear Attention Sequence Parallelism (LASP)
  ☆84Updated last year
• aykutcayir34 / DifferentialTransformer
  ☆13Updated 8 months ago
• ziplab / QLLM
  [ICLR 2024] This is the official PyTorch implementation of "QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Mod…
  ☆27Updated last year
• lhallee / Multi_Head_Mixture_of_Experts__MH-MOE
  ☆28Updated 8 months ago
• DACUS1995 / pytorch-mmap-dataset
  A custom pytorch Dataset extension that provides a faster iteration and better RAM usage
  ☆43Updated last year
• aniquetahir / JORA
  JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)
  ☆34Updated last year