uclaml / MoELinks
Towards Understanding the Mixture-of-Experts Layer in Deep Learning
☆31Updated last year
Alternatives and similar repositories for MoE
Users that are interested in MoE are comparing it to the libraries listed below
Sorting:
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆76Updated last year
- State Space Models☆70Updated last year
- HGRN2: Gated Linear RNNs with State Expansion☆55Updated 11 months ago
- Official repository of "LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging"☆30Updated 9 months ago
- Decomposing and Editing Predictions by Modeling Model Computation☆138Updated last year
- ☆44Updated 6 months ago
- ☆147Updated 11 months ago
- Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling☆201Updated 2 weeks ago
- Official Code for Paper: Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation☆116Updated last month
- A regression-alike loss to improve numerical reasoning in language models☆24Updated 3 weeks ago
- One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation☆41Updated 9 months ago
- Official Code for ICLR 2024 Paper: Non-negative Contrastive Learning☆45Updated last year
- Model Stock: All we need is just a few fine-tuned models☆121Updated 10 months ago
- ☆185Updated last year
- Code and benchmark for the paper: "A Practitioner's Guide to Continual Multimodal Pretraining" [NeurIPS'24]☆57Updated 7 months ago
- A repository for DenseSSMs☆88Updated last year
- ☆34Updated 7 months ago
- Awesome list of papers that extend Mamba to various applications.☆134Updated last month
- ☆30Updated 5 months ago
- Code for PHATGOOSE introduced in "Learning to Route Among Specialized Experts for Zero-Shot Generalization"☆86Updated last year
- A curated list of Model Merging methods.☆92Updated 10 months ago
- [NeurIPS 2024] RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models☆25Updated 8 months ago
- Official implementation of MAIA, A Multimodal Automated Interpretability Agent☆83Updated last month
- Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"☆56Updated 2 weeks ago
- Recycling diverse models☆45Updated 2 years ago
- Optimal Transport in the Big Data Era☆109Updated 9 months ago
- Official implementation of ORCA proposed in the paper "Cross-Modal Fine-Tuning: Align then Refine"☆71Updated last year
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆53Updated 2 years ago
- Code for NOLA, an implementation of "nola: Compressing LoRA using Linear Combination of Random Basis"☆55Updated 11 months ago
- ☆46Updated last year