YeonwooSung / Pytorch_mixture-of-expertsLinks
PyTorch implementation of moe, which stands for mixture of experts
☆50Updated 4 years ago
Alternatives and similar repositories for Pytorch_mixture-of-experts
Users that are interested in Pytorch_mixture-of-experts are comparing it to the libraries listed below
Sorting:
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆78Updated 2 years ago
- several types of attention modules written in PyTorch for learning purposes☆52Updated last year
- Implementation of Infini-Transformer in Pytorch☆113Updated 9 months ago
- Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts☆119Updated last year
- Implementation of MoE Mamba from the paper: "MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts" in Pytorch and Ze…☆113Updated last week
- Pytorch Implementation of the sparse attention from the paper: "Generating Long Sequences with Sparse Transformers"☆91Updated last week
- MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248☆57Updated last year
- Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"☆124Updated last year
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆100Updated last year
- This is a simple torch implementation of the high performance Multi-Query Attention☆15Updated 2 years ago
- A repository for DenseSSMs☆89Updated last year
- Implementation of TableFormer, Robust Transformer Modeling for Table-Text Encoding, in Pytorch☆39Updated 3 years ago
- Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind☆177Updated last year
- Implementation of Agent Attention in Pytorch☆91Updated last year
- Playground for Transformers☆53Updated last year
- A regression-alike loss to improve numerical reasoning in language models - ICML 2025☆26Updated 2 months ago
- Self contained pytorch implementation of a sinkhorn based router, for mixture of experts or otherwise☆39Updated last year
- Implementation of the general framework for AMIE, from the paper "Towards Conversational Diagnostic AI", out of Google Deepmind☆68Updated last year
- Code for NOLA, an implementation of "nola: Compressing LoRA using Linear Combination of Random Basis"☆56Updated last year
- This is a PyTorch implementation of the paperViP A Differentially Private Foundation Model for Computer Vision☆36Updated 2 years ago
- Towards Understanding the Mixture-of-Experts Layer in Deep Learning☆32Updated last year
- State Space Models☆70Updated last year
- ☆186Updated last year
- Model Stock: All we need is just a few fine-tuned models☆125Updated 2 months ago
- A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.☆96Updated 10 months ago
- ☆134Updated 2 years ago
- Timm model explorer☆42Updated last year
- The official implementation of the paper "What Matters in Transformers? Not All Attention is Needed".☆179Updated 7 months ago
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆129Updated last year
- code for the ddp tutorial☆32Updated 3 years ago