Alternatives and similar repositories for grouped-query-attention-pytorch:

Users that are interested in grouped-query-attention-pytorch are comparing it to the libraries listed below

• OpenNLPLab / lightning-attention
  Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  ☆282Updated 2 months ago
• Outsider565 / LoRA-GA
  ☆194Updated 6 months ago
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆91Updated last week
• OpenNLPLab / TransnormerLLM
  Official implementation of TransNormerLLM: A Faster and Better LLM
  ☆243Updated last year
• fxmeng / TransMLA
  TransMLA: Multi-Head Latent Attention Is All You Need
  ☆243Updated this week
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆158Updated 10 months ago
• nbasyl / DoRA
  Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
  ☆123Updated last year
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆128Updated last year
• fla-org / flame
  🔥 A minimal training framework for scaling FLA models
  ☆119Updated this week
• Clin0212 / HydraLoRA
  [NeurIPS'24 Oral] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
  ☆195Updated 5 months ago
• DRSY / EMO
  [ICLR 2024]EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling(https://arxiv.org/abs/2310.04691)
  ☆122Updated last year
• yxli2123 / LoftQ
  ☆220Updated 10 months ago
• transformer-vq / transformer_vq
  ☆189Updated last year
• shaochenze / PatchTrain
  Code for paper "Patch-Level Training for Large Language Models"
  ☆84Updated 5 months ago
• princeton-nlp / CEPE
  [ACL 2024] Long-Context Language Modeling with Parallel Encodings
  ☆154Updated 10 months ago
• lucidrains / soft-moe-pytorch
  Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch
  ☆287Updated last month
• QingruZhang / AdaLoRA
  AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).
  ☆320Updated last year
• GCYZSL / MoLA
  ☆132Updated 9 months ago
• Cohere-Labs-Community / parameter-efficient-moe
  ☆256Updated last year
• fkodom / yet-another-retnet
  A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (http…
  ☆105Updated last year
• lucidrains / st-moe-pytorch
  Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch
  ☆328Updated 10 months ago
• bojone / rerope
  Rectified Rotary Position Embeddings
  ☆367Updated 11 months ago
• haochengxi / Train_Transformers_with_INT4
  ☆147Updated last year
• DAMO-NLP-SG / Inf-CLIP
  [CVPR 2025 Highlight] The official CLIP training codebase of Inf-CL: "Breaking the Memory Barrier: Near Infinite Batch Size Scaling for C…
  ☆242Updated 3 months ago
• wutaiqiang / MoSLoRA
  ☆100Updated 10 months ago
• sramshetty / mixture-of-depths
  An unofficial implementation of "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆35Updated 11 months ago
• Ablustrund / LoRAMoE
  LoRAMoE: Revolutionizing Mixture of Experts for Maintaining World Knowledge in Language Model Alignment
  ☆327Updated last year
• facebookresearch / LLM-QAT
  Code repo for the paper "LLM-QAT Data-Free Quantization Aware Training for Large Language Models"
  ☆284Updated 2 months ago
• kyegomez / SwitchTransformers
  Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficien…
  ☆100Updated last month
• berlino / gated_linear_attention
  ☆103Updated last year