uclaml / MoELinks
Towards Understanding the Mixture-of-Experts Layer in Deep Learning
☆31Updated last year
Alternatives and similar repositories for MoE
Users that are interested in MoE are comparing it to the libraries listed below
Sorting:
- State Space Models☆70Updated last year
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆78Updated last year
- Decomposing and Editing Predictions by Modeling Model Computation☆138Updated last year
- Official repository of "LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging"☆30Updated 10 months ago
- ☆48Updated 7 months ago
- HGRN2: Gated Linear RNNs with State Expansion☆54Updated last year
- ☆32Updated 7 months ago
- A regression-alike loss to improve numerical reasoning in language models - ICML 2025☆25Updated last month
- Code and benchmark for the paper: "A Practitioner's Guide to Continual Multimodal Pretraining" [NeurIPS'24]☆58Updated 9 months ago
- Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling☆207Updated 2 weeks ago
- ☆148Updated last year
- ☆190Updated last year
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)☆31Updated 5 months ago
- One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation☆42Updated 11 months ago
- Unofficial Implementation of Selective Attention Transformer☆17Updated 10 months ago
- A curated list of Model Merging methods.☆92Updated last year
- ☆19Updated 5 months ago
- A repository for DenseSSMs☆88Updated last year
- ☆20Updated last year
- An official implementation of "Catastrophic Failure of LLM Unlearning via Quantization" (ICLR 2025)☆29Updated 7 months ago
- ☆30Updated last year
- Model Stock: All we need is just a few fine-tuned models☆124Updated last month
- source code for paper "Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models"☆31Updated last year
- Code for PHATGOOSE introduced in "Learning to Route Among Specialized Experts for Zero-Shot Generalization"☆88Updated last year
- ☆33Updated 8 months ago
- MultiModN – Multimodal, Multi-Task, Interpretable Modular Networks (NeurIPS 2023)☆33Updated last year
- Optimal Transport in the Big Data Era☆110Updated 10 months ago
- ☆85Updated last year
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆55Updated 2 years ago
- Official implementation of Vector-ICL: In-context Learning with Continuous Vector Representations (ICLR 2025)☆20Updated 3 months ago