Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
☆176Jun 20, 2024Updated 2 years ago
Alternatives and similar repositories for Mixture-of-depths
Users that are interested in Mixture-of-depths are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆124Oct 26, 2026Updated last week
- An unofficial implementation of "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆35Jun 7, 2024Updated 2 years ago
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆153Oct 15, 2024Updated 2 years ago
- Google DeepMind: Mixture of Depths Unofficial Implementation.☆12May 29, 2024Updated 2 years ago
- ☆15Apr 11, 2024Updated 2 years ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- [ICLR2025] γ -MOD: Mixture-of-Depth Adaptation for Multimodal Large Language Models☆46Oct 28, 2025Updated last year
- [ACL 24 Findings] Implementation of Resonance RoPE and the PosGen synthetic dataset.☆24Mar 5, 2024Updated 2 years ago
- My fork os allen AI's OLMo for educational purposes.☆29Dec 5, 2024Updated last year
- ☆173Feb 15, 2025Updated last year
- [ICML 2025] Code for "R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts"☆19Mar 10, 2025Updated last year
- Official repo of dataset-decomposition paper [NeurIPS 2024]☆21Sep 11, 2026Updated last month
- APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding☆14Jul 22, 2024Updated 2 years ago
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)☆166Apr 13, 2025Updated last year
- Evaluation code for RBench-V. Based on https://github.com/open-compass/VLMEvalKit.☆12May 24, 2025Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Source code of paper ''KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing''☆32Oct 24, 2024Updated 2 years ago
- [NeurIPS 2024] Official Repository of The Mamba in the Llama: Distilling and Accelerating Hybrid Models☆244Oct 14, 2025Updated last year
- ☆336Jul 10, 2025Updated last year
- Complex RAG backend☆29Mar 28, 2024Updated 2 years ago
- Adaptive Topology Reconstruction for Robust Graph Representation Learning [Efficient ML Model]☆10Feb 11, 2025Updated last year
- [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads☆545Feb 10, 2025Updated last year
- Mixture of A Million Experts☆64Jul 30, 2024Updated 2 years ago
- My Implementation of Q-Sparse: All Large Language Models can be Fully Sparsely-Activated☆36Aug 14, 2024Updated 2 years ago
- Official Repository for Paper "BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Compet…☆18Aug 28, 2024Updated 2 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆761Sep 27, 2024Updated 2 years ago
- Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)☆67Sep 28, 2024Updated 2 years ago
- Code and Dataset release of "Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models" (NAACL 2024)☆11Oct 16, 2024Updated 2 years ago
- The hearth of The Pulsar App, fast, secure and shared inference with modern UI☆58Dec 1, 2024Updated last year
- Unofficial implementations of block/layer-wise pruning methods for LLMs.☆78Apr 29, 2024Updated 2 years ago
- EE-LLM is a framework for large-scale training and inference of early-exit (EE) large language models (LLMs).☆83Jun 14, 2024Updated 2 years ago
- AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers☆48Oct 21, 2022Updated 4 years ago
- Official PyTorch Implementation of the Longhorn Deep State Space Model☆58Dec 4, 2024Updated last year
- ⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)☆1,004Dec 6, 2024Updated last year
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- [ICLR 2025] MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts☆281Oct 16, 2024Updated 2 years ago
- The Bytepiece Tokenizer Implemented in Rust.☆15Nov 28, 2023Updated 2 years ago
- A family of open-sourced Mixture-of-Experts (MoE) Large Language Models☆1,702Mar 8, 2024Updated 2 years ago
- Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport☆15Feb 26, 2025Updated last year
- A collection of AWESOME things about mixture-of-experts☆1,287Dec 8, 2024Updated last year
- Accelerating Multitask Training Trough Adaptive Transition [Efficient ML Model]☆12May 23, 2025Updated last year
- Efficient Foundation Model Design: A Perspective From Model and System Co-Design [Efficient ML System & Model]☆31Feb 23, 2025Updated last year