YeonwooSung / Pytorch_mixture-of-experts
PyTorch implementation of moe, which stands for mixture of experts
☆42Updated 4 years ago
Alternatives and similar repositories for Pytorch_mixture-of-experts:
Users that are interested in Pytorch_mixture-of-experts are comparing it to the libraries listed below
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆71Updated last year
- Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts☆118Updated 5 months ago
- Implementation of Infini-Transformer in Pytorch☆110Updated 2 months ago
- Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"☆123Updated 11 months ago
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆97Updated 6 months ago
- Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆86Updated 3 weeks ago
- 32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.☆47Updated last year
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)☆150Updated 3 months ago
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆122Updated 7 months ago
- Playground for Transformers☆48Updated last year
- Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind☆174Updated 6 months ago
- PyTorch implementation of "From Sparse to Soft Mixtures of Experts"☆53Updated last year
- Pytorch Implementation of the paper: "Learning to (Learn at Test Time): RNNs with Expressive Hidden States"☆24Updated 2 weeks ago
- Explorations into adversarial losses on top of autoregressive loss for language modeling☆35Updated last month
- Timm model explorer☆37Updated 11 months ago
- This is a PyTorch implementation of the paperViP A Differentially Private Foundation Model for Computer Vision☆36Updated last year
- Low-bit optimizers for PyTorch☆125Updated last year
- Implementation of a multimodal diffusion transformer in Pytorch☆101Updated 9 months ago
- Implementation of a Light Recurrent Unit in Pytorch☆47Updated 5 months ago
- imagetokenizer is a python package, helps you encoder visuals and generate visuals token ids from codebook, supports both image and video…☆31Updated 9 months ago
- A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.☆90Updated 3 months ago
- Fine-tuning LLaMA to follow Instructions within 1 Hour and 1.2M Parameters☆88Updated last year
- Collection of autoregressive model implementation☆83Updated last month
- Model Stock: All we need is just a few fine-tuned models☆107Updated 6 months ago
- Code for NeurIPS LLM Efficiency Challenge☆57Updated 11 months ago
- Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks☆141Updated 6 months ago
- Just some miscellaneous utility functions / decorators / modules related to Pytorch and Accelerate to help speed up implementation of new…☆120Updated 8 months ago
- code for the ddp tutorial☆32Updated 2 years ago
- One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation☆38Updated 5 months ago
- Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M cont…☆81Updated 10 months ago