Alternatives and similar repositories for FlashWindowAttention

Users that are interested in FlashWindowAttention are comparing it to the libraries listed below

• lucidrains / soft-moe-pytorch
  Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch
  ☆294Updated 2 months ago
• MzeroMiko / mamba-mini
  An efficient pytorch implementation of selective scan in one file, works with both cpu and gpu, with corresponding mathematical derivatio…
  ☆89Updated last year
• SteveTsui / Q-DETR
  ☆34Updated last year
• OscarXZQ / weight-selection
  ☆181Updated 8 months ago
• AmeenAli / HiddenMambaAttn
  Official PyTorch Implementation of "The Hidden Attention of Mamba Models"
  ☆222Updated last year
• bwconrad / soft-moe
  PyTorch implementation of "From Sparse to Soft Mixtures of Experts"
  ☆57Updated last year
• apple / ml-sigmoid-attention
  ☆286Updated last month
• berlino / gated_linear_attention
  ☆104Updated last year
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆94Updated this week
• WailordHe / DenseSSM
  A repository for DenseSSMs
  ☆87Updated last year
• fkodom / grouped-query-attention-pytorch
  (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …
  ☆168Updated last year
• TomerRonen34 / mixed-resolution-vit
  ☆50Updated last year
• OpenNLPLab / lightning-attention
  Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  ☆296Updated 3 months ago
• LeapLabTHU / EfficientTrain
  1.5−3.0× lossless training or pre-training speedup. An off-the-shelf, easy-to-implement algorithm for the efficient training of foundatio…
  ☆220Updated 9 months ago
• mit-han-lab / sparsevit
  [CVPR'23] SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer
  ☆69Updated last year
• fawazsammani / awesome-mlp-mixer
  Transformers w/o Attention, based fully on MLPs
  ☆93Updated last year
• karttikeya / minREV
  A simple minimal implementation of Reversible Vision Transformers
  ☆125Updated last year
• Hprairie / Bi-Mamba2
  A Triton Kernel for incorporating Bi-Directionality in Mamba2
  ☆68Updated 5 months ago
• naver-ai / rope-vit
  [ECCV 2024] Official PyTorch implementation of RoPE-ViT "Rotary Position Embedding for Vision Transformer"
  ☆327Updated 5 months ago
• lhallee / Multi_Head_Mixture_of_Experts__MH-MOE
  ☆28Updated 7 months ago
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆97Updated 8 months ago
• goombalab / hydra
  Official implementation of "Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers"
  ☆137Updated 4 months ago
• Hao840 / vanillaKD
  PyTorch code and checkpoints release for VanillaKD: https://arxiv.org/abs/2305.15781
  ☆75Updated last year
• FocalNet / FocalNet-DINO
  This repo contains the code and configuration files for reproducing object detection results of FocalNets with DINO
  ☆67Updated 2 years ago
• ziplab / EcoFormer
  [NeurIPS 2022 Spotlight] This is the official PyTorch implementation of "EcoFormer: Energy-Saving Attention with Linear Complexity"
  ☆72Updated 2 years ago
• changlin31 / AutoProg
  (CVPR 2022) Automated Progressive Learning for Efficient Training of Vision Transformers
  ☆25Updated 3 months ago
• eric-ai-lab / PEViT
  Official implementation of AAAI 2023 paper "Parameter-efficient Model Adaptation for Vision Transformers"
  ☆105Updated last year
• SLDGroup / LBP-WHT
  ☆11Updated last year
• zugexiaodui / torch_flops
  A library for calculating the FLOPs in the forward() process based on torch.fx
  ☆113Updated 2 months ago
• MengLcool / AdaViT
  [CVPR-22] This is the official implementation of the paper "Adavit: Adaptive vision transformers for efficient image recognition".
  ☆54Updated 2 years ago