Alternatives and similar repositories for Pytorch_mixture-of-experts

Users that are interested in Pytorch_mixture-of-experts are comparing it to the libraries listed below

• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆98Updated 8 months ago
• lucidrains / infini-transformer-pytorch
  Implementation of Infini-Transformer in Pytorch
  ☆111Updated 5 months ago
• fkodom / soft-mixture-of-experts
  PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
  ☆73Updated last year
• nbasyl / DoRA
  Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
  ☆123Updated last year
• naver-ai / model-stock
  Model Stock: All we need is just a few fine-tuned models
  ☆117Updated 9 months ago
• ml-jku / EVA
  One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
  ☆40Updated 8 months ago
• kyegomez / Blockwise-Parallel-Transformer
  32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.
  ☆48Updated 2 years ago
• OpenNLPLab / LASP
  Linear Attention Sequence Parallelism (LASP)
  ☆84Updated last year
• kamanphoebe / Look-into-MoEs
  [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models
  ☆52Updated 4 months ago
• sramshetty / mixture-of-depths
  An unofficial implementation of "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆35Updated last year
• HanGuo97 / lq-lora
  ☆126Updated last year
• lucidrains / AMIE-pytorch
  Implementation of the general framework for AMIE, from the paper "Towards Conversational Diagnostic AI", out of Google Deepmind
  ☆64Updated 9 months ago
• jlamprou / Infini-Attention
  Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M cont…
  ☆83Updated last year
• lucidrains / tableformer-pytorch
  Implementation of TableFormer, Robust Transformer Modeling for Table-Text Encoding, in Pytorch
  ☆39Updated 3 years ago
• TRI-ML / linear_open_lm
  A repository for research on medium sized language models.
  ☆76Updated last year
• uclaml / MoE
  Towards Understanding the Mixture-of-Experts Layer in Deep Learning
  ☆31Updated last year
• snu-mllab / Context-Memory
  Pytorch implementation for "Compressed Context Memory For Online Language Model Interaction" (ICLR'24)
  ☆61Updated last year
• VITA-Group / WeLore
  From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…
  ☆47Updated 2 months ago
• jongwooko / distillm-2
  Official PyTorch implementation of DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (ICML 2025 Oral)
  ☆22Updated 2 weeks ago
• CASE-Lab-UMD / LLM-Drop
  The official implementation of the paper "What Matters in Transformers? Not All Attention is Needed".
  ☆173Updated 2 months ago
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆95Updated 2 weeks ago
• microsoft / AutoMoE
  AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
  ☆47Updated 2 years ago
• knotgrass / attention
  several types of attention modules written in PyTorch for learning purposes
  ☆52Updated 8 months ago
• r-three / phatgoose
  Code for PHATGOOSE introduced in "Learning to Route Among Specialized Experts for Zero-Shot Generalization"
  ☆85Updated last year
• wuhy68 / Parameter-Efficient-MoE
  Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  ☆144Updated 9 months ago
• Lichang-Chen / AlpaGasus
  A better Alpaca Model Trained with Less Data (only 9k instructions of the original set)
  ☆24Updated 10 months ago
• SeunghyunSEO / optimized_hf_llama_class_for_training
  ☆47Updated 9 months ago
• facebookresearch / ViP-MAE
  This is a PyTorch implementation of the paperViP A Differentially Private Foundation Model for Computer Vision
  ☆36Updated last year
• wang-kee / LiNeS
  Official repository of "LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging"
  ☆29Updated 7 months ago
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆52Updated 2 years ago