SimiaoZuo / MoEBERT
This PyTorch package implements MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation (NAACL 2022).
☆103Updated 2 years ago
Alternatives and similar repositories for MoEBERT:
Users that are interested in MoEBERT are comparing it to the libraries listed below
- ☆129Updated 8 months ago
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆50Updated 2 years ago
- This pytorch package implements PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance (ICML 2022).☆44Updated 2 years ago
- [ACL 2024] Long-Context Language Modeling with Parallel Encodings☆153Updated 10 months ago
- ☆98Updated 6 months ago
- [ACL 2022] Structured Pruning Learns Compact and Accurate Models https://arxiv.org/abs/2204.00408☆195Updated last year
- This package implements THOR: Transformer with Stochastic Experts.☆61Updated 3 years ago
- Implementation of ICML 23 Paper: Specializing Smaller Language Models towards Multi-Step Reasoning.☆130Updated last year
- Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)☆57Updated 6 months ago
- Codes for our paper "Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation" (EMNLP 2023 Findings)☆39Updated last year
- Code for the ACL-2022 paper "StableMoE: Stable Routing Strategy for Mixture of Experts"☆45Updated 2 years ago
- ACL'23: Unified Demonstration Retriever for In-Context Learning☆37Updated last year
- ☆48Updated last year
- ICML'2022: Black-Box Tuning for Language-Model-as-a-Service & EMNLP'2022: BBTv2: Towards a Gradient-Free Future with Large Language Model…☆267Updated 2 years ago
- Retrieval as Attention☆83Updated 2 years ago
- DSIR large-scale data selection framework for language model training☆245Updated last year
- Must-read papers on improving efficiency for pre-trained language models.☆103Updated 2 years ago
- [KDD'22] Learned Token Pruning for Transformers☆95Updated 2 years ago
- One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning☆39Updated last year
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models☆76Updated last year
- Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models☆140Updated 2 years ago
- Code associated with the paper **Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding**☆181Updated 2 months ago
- [NeurIPS'23] Speculative Decoding with Big Little Decoder☆90Updated last year
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)☆101Updated last month
- Code for ACL2023 paper: Pre-Training to Learn in Context☆108Updated 8 months ago
- ☆39Updated last year
- [ICLR 2025] 🧬 RegMix: Data Mixture as Regression for Language Model Pre-training (Spotlight)☆124Updated 2 months ago
- Official PyTorch implementation of DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024)☆207Updated last month
- Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method ; GKD: A General Knowledge Distillation…☆32Updated last year
- This is the oficial repository for "Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts" (EMNLP 2022)☆100Updated 2 years ago