facebookresearch / Mixture-of-Transformers
Mixture-of-Transformers A Sparse and Scalable Architecture for Multi-Modal Foundation Models. TMLR 2025. 🔗 https//arxiv.org/abs/2411.04996
☆31Updated this week
Alternatives and similar repositories for Mixture-of-Transformers
Users that are interested in Mixture-of-Transformers are comparing it to the libraries listed below
Sorting:
- One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation☆39Updated 6 months ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆57Updated 8 months ago
- ☆24Updated 3 months ago
- The official repository for SkyLadder: Better and Faster Pretraining via Context Window Scheduling☆29Updated last month
- Code, results and other artifacts from the paper introducing the WildChat-50m dataset and the Re-Wild model family.☆29Updated last month
- Code for "Accelerating Training with Neuron Interaction and Nowcasting Networks" [to appear at ICLR 2025]☆19Updated 2 months ago
- Code and pretrained models for the paper: "MatMamba: A Matryoshka State Space Model"☆59Updated 5 months ago
- Synthetic data generation and benchmark implementation for "Episodic Memories Generation and Evaluation Benchmark for Large Language Mode…☆41Updated 3 weeks ago
- Official Repo for InSTA: Towards Internet-Scale Training For Agents☆36Updated 2 weeks ago
- Triton Implementation of HyperAttention Algorithm☆48Updated last year
- Repository for the Q-Filters method (https://arxiv.org/pdf/2503.02812)☆30Updated 2 months ago
- Exploration of automated dataset selection approaches at large scales.☆40Updated 2 months ago
- Latest Weight Averaging (NeurIPS HITY 2022)☆30Updated last year
- ☆31Updated 4 months ago
- ☆29Updated last year
- AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories☆12Updated 3 weeks ago
- Aioli: A unified optimization framework for language model data mixing☆25Updated 3 months ago
- Utilities for Training Very Large Models☆58Updated 7 months ago
- [NeurIPS 2024] Goldfish Loss: Mitigating Memorization in Generative LLMs☆84Updated 5 months ago
- PyTorch library for Active Fine-Tuning☆72Updated 2 months ago
- ☆72Updated last year
- ☆19Updated 10 months ago
- Stanford NLP Python library for benchmarking the utility of LLM interpretability methods☆77Updated last month
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆72Updated 6 months ago
- Simple and scalable tools for data-driven pretraining data selection.☆23Updated 3 months ago
- This repo is based on https://github.com/jiaweizzhao/GaLore☆27Updated 7 months ago
- ☆68Updated 10 months ago
- ☆17Updated 3 months ago
- Maya: An Instruction Finetuned Multilingual Multimodal Model using Aya☆108Updated 2 months ago
- ReBase: Training Task Experts through Retrieval Based Distillation☆29Updated 3 months ago