Alternatives and similar repositories for grouped-query-attention-pytorch

Users that are interested in grouped-query-attention-pytorch are comparing it to the libraries listed below

• OpenNLPLab / lightning-attention
  Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  ☆294Updated 3 months ago
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆94Updated this week
• Outsider565 / LoRA-GA
  ☆198Updated 7 months ago
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆161Updated 11 months ago
• nbasyl / DoRA
  Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
  ☆123Updated last year
• shaochenze / PatchTrain
  Code for paper "Patch-Level Training for Large Language Models"
  ☆86Updated 6 months ago
• transformer-vq / transformer_vq
  ☆191Updated last year
• pprp / Awesome-Efficient-MoE
  Efficient Mixture of Experts for LLM Paper List
  ☆68Updated 5 months ago
• OpenNLPLab / TransnormerLLM
  Official implementation of TransNormerLLM: A Faster and Better LLM
  ☆243Updated last year
• yxli2123 / LoftQ
  ☆222Updated 11 months ago
• YuchuanTian / RethinkTinyLM
  [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”
  ☆121Updated 4 months ago
• fla-org / flame
  🔥 A minimal training framework for scaling FLA models
  ☆146Updated 3 weeks ago
• bzhangGo / rmsnorm
  Root Mean Square Layer Normalization
  ☆241Updated 2 years ago
• Cohere-Labs-Community / parameter-efficient-moe
  ☆258Updated last year
• ZhenweiAn / Dynamic_MoE
  Inference Code for Paper "Harder Tasks Need More Experts: Dynamic Routing in MoE Models"
  ☆50Updated 10 months ago
• bojone / rerope
  Rectified Rotary Position Embeddings
  ☆370Updated last year
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆128Updated last year
• berlino / gated_linear_attention
  ☆103Updated last year
• fxmeng / TransMLA
  TransMLA: Multi-Head Latent Attention Is All You Need
  ☆284Updated this week
• lucidrains / soft-moe-pytorch
  Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch
  ☆294Updated 2 months ago
• CASIA-IVA-Lab / FLAP
  [AAAI 2024] Fluctuation-based Adaptive Structured Pruning for Large Language Models
  ☆51Updated last year
• mdy666 / Qwen-Native-Sparse-Attention
  qwen-nsa
  ☆66Updated last month
• princeton-nlp / CEPE
  [ACL 2024] Long-Context Language Modeling with Parallel Encodings
  ☆153Updated 11 months ago
• thu-ml / ReMoE
  [ICLR2025] Codebase for "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing", built on Megatron-LM.
  ☆74Updated 5 months ago
• NVlabs / COAT
  [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Training
  ☆203Updated 2 weeks ago
• haochengxi / Train_Transformers_with_INT4
  ☆149Updated last year
• shreyansh26 / FlashAttention-PyTorch
  Implementation of FlashAttention in PyTorch
  ☆150Updated 4 months ago
• XunhaoLai / native-sparse-attention-triton
  Efficient triton implementation of Native Sparse Attention.
  ☆155Updated last week
• dhcode-cpp / NSA-pytorch
  DeepSeek Native Sparse Attention pytorch implementation
  ☆70Updated 3 months ago
• ylsung / Ladder-Side-Tuning
  PyTorch codes for "LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning"
  ☆238Updated 2 years ago