goombalab / phi-mambaLinks
Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models)
☆108Updated 8 months ago
Alternatives and similar repositories for phi-mamba
Users that are interested in phi-mamba are comparing it to the libraries listed below
Sorting:
- [NeurIPS 2024] Official Repository of The Mamba in the Llama: Distilling and Accelerating Hybrid Models☆221Updated last month
- Stick-breaking attention☆56Updated 2 months ago
- Some preliminary explorations of Mamba's context scaling.☆212Updated last year
- Official implementation of "Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers"☆135Updated 4 months ago
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆234Updated 3 months ago
- Inference Speed Benchmark for Learning to (Learn at Test Time): RNNs with Expressive Hidden States☆67Updated 10 months ago
- ☆79Updated 9 months ago
- Code accompanying the paper "Massive Activations in Large Language Models"☆162Updated last year
- Understand and test language model architectures on synthetic tasks.☆197Updated 2 months ago
- ☆54Updated 10 months ago
- A MAD laboratory to improve AI architecture designs 🧪☆116Updated 5 months ago
- [ICLR 2025] Official PyTorch Implementation of Gated Delta Networks: Improving Mamba2 with Delta Rule☆167Updated 2 months ago
- Official PyTorch Implementation of "The Hidden Attention of Mamba Models"☆222Updated last year
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆124Updated 9 months ago
- [ICLR 2025] Official PyTorch implementation of "Forgetting Transformer: Softmax Attention with a Forget Gate"☆104Updated 2 weeks ago
- [ICML 2025] Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization☆69Updated 4 months ago
- ☆92Updated 8 months ago
- [ICLR 2025] Code for the paper "Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning"☆59Updated 3 months ago
- ☆47Updated 2 months ago
- [NeurIPS 2024] Code for the paper "Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models"☆153Updated 2 months ago
- Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch☆170Updated 5 months ago
- ☆74Updated 3 months ago
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆126Updated 3 weeks ago
- [ICLR2025] Codebase for "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing", built on Megatron-LM.☆74Updated 5 months ago
- Official Implementation for the paper "d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning"☆164Updated 3 weeks ago
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)☆28Updated last month
- ☆52Updated last year
- Code for "Reasoning to Learn from Latent Thoughts"☆104Updated 2 months ago
- [ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)☆84Updated 7 months ago
- Official PyTorch Implementation of the Longhorn Deep State Space Model☆50Updated 6 months ago