junfanz1 / MoE-Mixture-of-Experts-in-PyTorchLinks
Implementations of a Mixture-of-Experts (MoE) architecture designed for research on large language models (LLMs) and scalable neural network designs. One implementation targets a **single-device/NPU environment** while the other is built for multi-device distributed computing. Both versions showcase the core principles.
☆36Updated 8 months ago
Alternatives and similar repositories for MoE-Mixture-of-Experts-in-PyTorch
Users that are interested in MoE-Mixture-of-Experts-in-PyTorch are comparing it to the libraries listed below
Sorting:
- PyTorch Implementation of Jamba: "Jamba: A Hybrid Transformer-Mamba Language Model"☆201Updated last month
- (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …☆186Updated last year
- Implementation of Infini-Transformer in Pytorch☆113Updated 11 months ago
- Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch☆374Updated last year
- Experiments on Multi-Head Latent Attention☆99Updated last year
- Enable Next-sentence Prediction for Large Language Models with Faster Speed, Higher Accuracy and Longer Context☆41Updated last year
- Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"☆56Updated last month
- First-principle implementations of groundbreaking AI algorithms using a wide range of deep learning frameworks, accompanied by supporting…☆180Updated 5 months ago
- Direct Preference Optimization from scratch in PyTorch☆122Updated 8 months ago
- The official implementation of the paper "What Matters in Transformers? Not All Attention is Needed".☆186Updated last month
- Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficien…☆135Updated last month
- minimal GRPO implementation from scratch☆100Updated 9 months ago
- An extension of the nanoGPT repository for training small MOE models.☆219Updated 9 months ago
- Notes on the Mamba and the S4 model (Mamba: Linear-Time Sequence Modeling with Selective State Spaces)☆175Updated last year
- ☆99Updated last year
- PyTorch implementation of the Differential-Transformer architecture for sequence modeling, specifically tailored as a decoder-only model …☆84Updated last year
- [ICLR 2024]EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling(https://arxiv.org/abs/2310.04691)☆127Updated last year
- It is a comprehensive resource hub compiling all LLM papers accepted at the International Conference on Learning Representations (ICLR) i…☆67Updated last year
- A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (http…☆106Updated 2 years ago
- Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆176Updated last year
- A repository for DenseSSMs☆89Updated last year
- Survey of Small Language Models from Penn State, ...☆229Updated last month
- LORA: Low-Rank Adaptation of Large Language Models implemented using PyTorch☆118Updated 2 years ago
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆81Updated 2 years ago
- Notes and commented code for RLHF (PPO)☆120Updated last year
- Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling☆211Updated 2 months ago
- Distributed training (multi-node) of a Transformer model☆90Updated last year
- ☆303Updated 8 months ago
- Awesome Large Vision-Language Model: A Curated List of Large Vision-Language Model☆39Updated 5 months ago
- several types of attention modules written in PyTorch for learning purposes☆52Updated last year