Alternatives and similar repositories for MoE:

Users that are interested in MoE are comparing it to the libraries listed below

• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆52Updated 4 months ago
• uclaml / PDE
  Official repo of Progressive Data Expansion: data, code and evaluation
  ☆27Updated last year
• wang-kee / LiNeS
  ☆23Updated 2 months ago
• peterljq / Parsimonious-Concept-Engineering
  Implementation of PaCE: Parsimonious Concept Engineering for Large Language Models (NeurIPS 2024)
  ☆31Updated 2 months ago
• LZY-the-boys / Twin-Merging
  [NeurIPS2024] Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging
  ☆47Updated last month
• formll / resolving-scaling-law-discrepancies
  ☆16Updated 6 months ago
• yidingjiang / ado
  The repository contains code for Adaptive Data Optimization
  ☆21Updated last month
• ablghtianyi / ICL_Modular_Arithmetic
  ☆16Updated 2 months ago
• nverma1 / merging-text-transformers
  Code for "Merging Text Transformers from Different Initializations"
  ☆19Updated 5 months ago
• twinkle0331 / LGTM
  [ACL 2023] Code for paper “Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation”(https://arxiv.org/abs/2305.…
  ☆38Updated last year
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆48Updated last year
• csinva / tree-prompt
  Tree prompting: easy-to-use scikit-learn interface for improved prompting.
  ☆35Updated last year
• JeanKaddour / NoTrainNoGain
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆80Updated last year
• facebookresearch / ModelRatatouille
  Recycling diverse models
  ☆44Updated 2 years ago
• r-three / phatgoose
  Code for PHATGOOSE introduced in "Learning to Route Among Specialized Experts for Zero-Shot Generalization"
  ☆80Updated 10 months ago
• shangdatalab / Deep-Contam
  ☆10Updated 4 months ago
• ml-jku / EVA
  One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
  ☆36Updated 3 months ago
• fkodom / soft-mixture-of-experts
  PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
  ☆69Updated last year
• tml-epfl / why-weight-decay
  Why Do We Need Weight Decay in Modern Deep Learning? [NeurIPS 2024]
  ☆58Updated 3 months ago
• maszhongming / ParaKnowTransfer
  Code for "Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective"
  ☆31Updated 8 months ago
• VijayLingam95 / SVFT
  ☆26Updated 7 months ago
• EnnengYang / AdaMerging
  AdaMerging: Adaptive Model Merging for Multi-Task Learning. ICLR, 2024.
  ☆61Updated 2 months ago
• IBM / ColPret
  Efficient Scaling laws and collaborative pretraining.
  ☆13Updated 2 months ago
• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆51Updated 2 months ago
• RobertCsordas / moeut
  ☆69Updated 5 months ago
• PKU-ML / non_neg
  Official Code for ICLR 2024 Paper: Non-negative Contrastive Learning
  ☆38Updated 8 months ago
• ExplainableML / fomo_in_flux
  Code and benchmark for the paper: "A Practitioner's Guide to Continual Multimodal Pretraining" [NeurIPS'24]
  ☆47Updated last month
• zzwjames / FailureLLMUnlearning
  ☆23Updated last month
• rtaori / data_feedback
  Code for the paper "Data Feedback Loops: Model-driven Amplification of Dataset Biases"
  ☆15Updated 2 years ago
• assafbk / DeciMamba
  Official Implementation Of The Paper: `DeciMamba: Exploring the Length Extrapolation Potential of Mamba'
  ☆22Updated 5 months ago