Alternatives and similar repositories for MoE:

Users that are interested in MoE are comparing it to the libraries listed below

• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆54Updated 8 months ago
• wang-kee / LiNeS
  Official repository of "LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging"
  ☆25Updated 6 months ago
• fkodom / soft-mixture-of-experts
  PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
  ☆71Updated last year
• ExplainableML / fomo_in_flux
  Code and benchmark for the paper: "A Practitioner's Guide to Continual Multimodal Pretraining" [NeurIPS'24]
  ☆54Updated 4 months ago
• chuanyang-Zheng / DAPE
  The this is the official implementation of "DAPE: Data-Adaptive Positional Encoding for Length Extrapolation"
  ☆37Updated 6 months ago
• PKU-ML / non_neg
  Official Code for ICLR 2024 Paper: Non-negative Contrastive Learning
  ☆45Updated last year
• EPFLiGHT / MultiModN
  MultiModN – Multimodal, Multi-Task, Interpretable Modular Networks (NeurIPS 2023)
  ☆33Updated last year
• formll / resolving-scaling-law-discrepancies
  ☆18Updated 9 months ago
• gstoica27 / KnOTS
  Model Merging with SVD to Tie the KnOTS [ICLR 2025]
  ☆52Updated last month
• Bradley-Butcher / Conformers
  Unofficial implementation of Conformal Language Modeling by Quach et al
  ☆28Updated last year
• facebookresearch / ModelRatatouille
  Recycling diverse models
  ☆44Updated 2 years ago
• ablghtianyi / ICL_Modular_Arithmetic
  ☆18Updated last month
• tanganke / weight-ensembling_MoE
  Code for paper "Merging Multi-Task Models via Weight-Ensembling Mixture of Experts"
  ☆24Updated 10 months ago
• Leiay / looped_transformer
  ☆25Updated last year
• Christina200 / Online-LoRA-official
  [WACV 2025] Official implementation of "Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation" by Xiwen Wei, Guihong L…
  ☆35Updated 5 months ago
• fredzzhang / atlas
  Official PyTorch implementation for NeurIPS'24 paper "Knowledge Composition using Task Vectors with Learned Anisotropic Scaling"
  ☆19Updated 2 months ago
• peterljq / Parsimonious-Concept-Engineering
  PaCE: Parsimonious Concept Engineering for Large Language Models (NeurIPS 2024)
  ☆35Updated 6 months ago
• uclaml / PDE
  Official repo of Progressive Data Expansion: data, code and evaluation
  ☆28Updated last year
• sjunhongshen / ORCA
  Official implementation of ORCA proposed in the paper "Cross-Modal Fine-Tuning: Align then Refine"
  ☆71Updated last year
• nverma1 / merging-text-transformers
  Code for "Merging Text Transformers from Different Initializations"
  ☆20Updated 3 months ago
• katiekang1998 / reasoning_generalization
  ☆31Updated 3 months ago
• Itamarzimm / UnifiedImplicitAttnRepr
  [ICLR 2025] Official Code Release for Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation
  ☆42Updated 2 months ago
• tml-epfl / why-weight-decay
  Why Do We Need Weight Decay in Modern Deep Learning? [NeurIPS 2024]
  ☆66Updated 7 months ago
• anniesch / surgical-finetuning
  Code for "Surgical Fine-Tuning Improves Adaptation to Distribution Shifts" published at ICLR 2023
  ☆29Updated last year
• WailordHe / DenseSSM
  A repository for DenseSSMs
  ☆87Updated last year
• assafbk / DeciMamba
  DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)
  ☆26Updated 3 weeks ago
• VijayLingam95 / SVFT
  ☆28Updated 2 months ago
• NuoJohnChen / JudgeLRM
  ☆24Updated 3 weeks ago
• UCDvision / NOLA
  Code for NOLA, an implementation of "nola: Compressing LoRA using Linear Combination of Random Basis"
  ☆54Updated 8 months ago
• twinkle0331 / LGTM
  [ACL 2023] Code for paper “Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation”(https://arxiv.org/abs/2305.…
  ☆38Updated last year