lucidrains / soft-moe-pytorch
Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch
☆260Updated 9 months ago
Alternatives and similar repositories for soft-moe-pytorch:
Users that are interested in soft-moe-pytorch are comparing it to the libraries listed below
- Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch☆306Updated 7 months ago
- When it comes to optimizers, it's always better to be safe than sorry☆179Updated 3 weeks ago
- Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts☆114Updated 3 months ago
- Some preliminary explorations of Mamba's context scaling.☆213Updated last year
- The official CLIP training codebase of Inf-CL: "Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss". A su…☆222Updated 3 weeks ago
- Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT☆206Updated 5 months ago
- Official code for "TOAST: Transfer Learning via Attention Steering"☆187Updated last year
- [ECCV 2024] Official PyTorch implementation of RoPE-ViT "Rotary Position Embedding for Vision Transformer"☆280Updated last month
- Implementation of Infini-Transformer in Pytorch☆109Updated last month
- ☆251Updated 5 months ago
- Language Quantized AutoEncoders☆99Updated 2 years ago
- Official implementation of "Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers"☆122Updated 2 weeks ago
- When do we not need larger vision models?☆364Updated last week
- Implementation of a multimodal diffusion transformer in Pytorch☆99Updated 7 months ago
- Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch☆403Updated last month
- ☆174Updated 4 months ago
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆71Updated last year
- Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…☆96Updated 5 months ago
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆220Updated 2 months ago
- Awesome list of papers that extend Mamba to various applications.☆131Updated last month
- Implementation of TiTok, proposed by Bytedance in "An Image is Worth 32 Tokens for Reconstruction and Generation"☆167Updated 7 months ago
- Just some miscellaneous utility functions / decorators / modules related to Pytorch and Accelerate to help speed up implementation of new…☆120Updated 6 months ago
- Official PyTorch Implementation of "Learning to Learn with Generative Models of Neural Network Checkpoints"☆337Updated 2 years ago
- [ICLR2025 Spotlight🔥] Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters☆495Updated this week
- A framework for merging models solving different tasks with different initializations into one multi-task model without any additional tr…☆292Updated last year
- Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch☆152Updated last month
- Implementation of Zorro, Masked Multimodal Transformer, in Pytorch☆95Updated last year
- Official code for our CVPR'22 paper “Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space”☆248Updated last year
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆117Updated 5 months ago
- An unofficial implementation of both ViT-VQGAN and RQ-VAE in Pytorch☆296Updated last year