junfanz1 / MoE-Mixture-of-Experts-in-PyTorchLinks
Implementations of a Mixture-of-Experts (MoE) architecture designed for research on large language models (LLMs) and scalable neural network designs. One implementation targets a **single-device/NPU environment** while the other is built for multi-device distributed computing. Both versions showcase the core principles.
☆41Updated 9 months ago
Alternatives and similar repositories for MoE-Mixture-of-Experts-in-PyTorch
Users that are interested in MoE-Mixture-of-Experts-in-PyTorch are comparing it to the libraries listed below
Sorting:
- PyTorch implementation of the Differential-Transformer architecture for sequence modeling, specifically tailored as a decoder-only model …☆85Updated last year
- Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficien…☆135Updated last week
- Awesome list of papers that extend Mamba to various applications.☆138Updated 7 months ago
- Research projects built on top of Transformers☆110Updated 10 months ago
- First-principle implementations of groundbreaking AI algorithms using a wide range of deep learning frameworks, accompanied by supporting…☆181Updated 5 months ago
- Official PyTorch Implementation of "The Hidden Attention of Mamba Models"☆231Updated 3 months ago
- (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …☆188Updated last year
- Notes on the Mamba and the S4 model (Mamba: Linear-Time Sequence Modeling with Selective State Spaces)☆178Updated 2 years ago
- PyTorch Implementation of Jamba: "Jamba: A Hybrid Transformer-Mamba Language Model"☆203Updated last week
- Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch☆375Updated last year
- Pytorch Implementation of the sparse attention from the paper: "Generating Long Sequences with Sparse Transformers"☆93Updated this week
- Mixture of Experts from scratch☆12Updated last year
- A easy, reliable, fluid template for python packages complete with docs, testing suites, readme's, github workflows, linting and much muc…☆199Updated this week
- Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025)☆530Updated 3 months ago
- Experiments on Multi-Head Latent Attention☆99Updated last year
- A repository for DenseSSMs☆88Updated last year
- ☆78Updated 7 months ago
- Reading list for research topics in state-space models☆341Updated 7 months ago
- minimal GRPO implementation from scratch☆102Updated 10 months ago
- ☆78Updated 11 months ago
- Conference schedule, top papers, and analysis of the data for NeurIPS 2023!☆120Updated 2 years ago
- Awesome Large Vision-Language Model: A Curated List of Large Vision-Language Model☆39Updated 5 months ago
- An extension of the nanoGPT repository for training small MOE models.☆225Updated 10 months ago
- Distributed training (multi-node) of a Transformer model☆91Updated last year
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆81Updated 2 years ago
- Reproduction of DeepSeek-R1☆242Updated 9 months ago
- ☆304Updated 8 months ago
- Explainable_AI_From_Classical_Models_to_LLMs Code☆48Updated 10 months ago
- Enable Next-sentence Prediction for Large Language Models with Faster Speed, Higher Accuracy and Longer Context☆41Updated last year
- ☆152Updated last year