Weixin-Liang / Mixture-of-Mamba
☆38Updated 2 months ago
Alternatives and similar repositories for Mixture-of-Mamba:
Users that are interested in Mixture-of-Mamba are comparing it to the libraries listed below
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)☆24Updated 8 months ago
- HGRN2: Gated Linear RNNs with State Expansion☆53Updated 7 months ago
- Official PyTorch Implementation for Task Vectors are Cross-Modal☆22Updated 3 months ago
- ☆35Updated last month
- We study toy models of skill learning.☆24Updated 2 months ago
- MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248☆51Updated 9 months ago
- More dimensions = More fun☆21Updated 8 months ago
- Code and benchmark for the paper: "A Practitioner's Guide to Continual Multimodal Pretraining" [NeurIPS'24]☆54Updated 3 months ago
- Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆86Updated 2 weeks ago
- A repository for DenseSSMs☆87Updated 11 months ago
- Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…☆102Updated 6 months ago
- Unofficial Implementation of Selective Attention Transformer☆16Updated 4 months ago
- Pytorch Implementation of the paper: "Learning to (Learn at Test Time): RNNs with Expressive Hidden States"☆24Updated last week
- The this is the official implementation of "DAPE: Data-Adaptive Positional Encoding for Length Extrapolation"☆37Updated 5 months ago
- ☆74Updated 7 months ago
- ☆43Updated 2 months ago
- A single repo with all scripts and utils to train / fine-tune the Mamba model with or without FIM☆54Updated 11 months ago
- Code for "R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts"☆14Updated 2 weeks ago
- RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best…☆42Updated last week
- ☆54Updated last month
- [ICLR 2025] Official PyTorch Implementation of Gated Delta Networks: Improving Mamba2 with Delta Rule☆143Updated last week
- Minimal Implementation of Visual Autoregressive Modelling (VAR)☆28Updated last week
- Auto Interpretation Pipeline and many other functionalities for Multimodal SAE Analysis.☆124Updated 2 months ago
- ☆48Updated 11 months ago
- State Space Models☆67Updated 10 months ago
- Implementation of MambaFormer in Pytorch ++ Zeta from the paper: "Can Mamba Learn How to Learn? A Comparative Study on In-Context Learnin…☆20Updated last week
- ☆39Updated 9 months ago
- Code for Heima☆37Updated last month
- [ICLR 2025] Official Code Release for Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation☆41Updated 3 weeks ago
- Official implementation of RMoE (Layerwise Recurrent Router for Mixture-of-Experts)☆18Updated 7 months ago