Alternatives and similar repositories for Pytorch_mixture-of-experts

Users that are interested in Pytorch_mixture-of-experts are comparing it to the libraries listed below

• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆97Updated 7 months ago
• itsnamgyu / block-transformer
  Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
  ☆154Updated last month
• naver-ai / model-stock
  Model Stock: All we need is just a few fine-tuned models
  ☆114Updated 7 months ago
• nbasyl / DoRA
  Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
  ☆123Updated last year
• jongwooko / distillm-2
  ☆13Updated 2 months ago
• lucidrains / infini-transformer-pytorch
  Implementation of Infini-Transformer in Pytorch
  ☆111Updated 4 months ago
• fkodom / soft-mixture-of-experts
  PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
  ☆72Updated last year
• kyegomez / Blockwise-Parallel-Transformer
  32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.
  ☆47Updated last year
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆91Updated this week
• timinar / BabyLlama
  Training code for Baby-Llama, our submission to the strict-small track of the BabyLM challenge.
  ☆80Updated last year
• lucidrains / transformer-lm-gan
  Explorations into adversarial losses on top of autoregressive loss for language modeling
  ☆36Updated 2 months ago
• SeunghyunSEO / optimized_hf_llama_class_for_training
  ☆47Updated 8 months ago
• YuchuanTian / RethinkTinyLM
  [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”
  ☆121Updated 4 months ago
• kyegomez / SimpleMamba
  Implementation of a modular, high-performance, and simplistic mamba for high-speed applications
  ☆34Updated 6 months ago
• lucidrains / light-recurrent-unit-pytorch
  Implementation of a Light Recurrent Unit in Pytorch
  ☆46Updated 7 months ago
• OscarXZQ / weight-selection
  ☆180Updated 7 months ago
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆118Updated 7 months ago
• ambisinister / mla-experiments
  Experiments on Multi-Head Latent Attention
  ☆89Updated 9 months ago
• Victorwz / LLaVA-Llama-3
  Reproduction of LLaVA-v1.5 based on Llama-3-8b LLM backbone.
  ☆65Updated 6 months ago
• sail-sg / scaling-with-vocab
  [NeurIPS-2024] 📈 Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies https://arxiv.org/abs/2407.13623
  ☆84Updated 7 months ago
• lucidrains / PEER-pytorch
  Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind
  ☆124Updated 8 months ago
• albanie / foundation-models
  Video descriptions of research papers relating to foundation models and scaling
  ☆31Updated 2 years ago
• wuhy68 / Parameter-Efficient-MoE
  Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  ☆143Updated 7 months ago
• HanGuo97 / lq-lora
  ☆125Updated last year
• ShiZhengyan / InstructionModelling
  [NeurIPS 2024 Main Track] Code for the paper titled "Instruction Tuning With Loss Over Instructions"
  ☆36Updated 11 months ago
• OpenNLPLab / LASP
  Linear Attention Sequence Parallelism (LASP)
  ☆82Updated 11 months ago
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆51Updated 2 years ago
• UbiquitousLearning / SLM_Survey
  ☆92Updated 7 months ago
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆128Updated last year
• fangyuan-ksgk / Mini-LLaVA
  A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.
  ☆91Updated 5 months ago