CASE-Lab-UMD / LLM-Drop
The official implementation of the paper "What Matters in Transformers? Not All Attention is Needed".
☆128Updated 2 weeks ago
Related projects ⓘ
Alternatives and complementary repositories for LLM-Drop
- Official PyTorch implementation of DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024)☆133Updated last month
- Code accompanying the paper "Massive Activations in Large Language Models"☆121Updated 8 months ago
- ☆197Updated 5 months ago
- Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks☆129Updated last month
- [NeurIPS 24 Spotlight] MaskLLM: Learnable Semi-structured Sparsity for Large Language Models☆108Updated last week
- ☆121Updated 9 months ago
- ☆149Updated 9 months ago
- The official implementation of the paper "Demystifying the Compression of Mixture-of-Experts Through a Unified Framework".☆48Updated 2 weeks ago
- ☆245Updated last year
- Prune transformer layers☆63Updated 5 months ago
- [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"☆64Updated 5 months ago
- ☆182Updated 3 weeks ago
- Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆134Updated 4 months ago
- Simple implementation of Speculative Sampling in NumPy for GPT-2.☆89Updated last year
- ☆198Updated 4 months ago
- Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)☆52Updated last month
- LLM-Merging: Building LLMs Efficiently through Merging☆174Updated last month
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆91Updated last month
- Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"☆122Updated 6 months ago
- Explorations into some recent techniques surrounding speculative decoding☆210Updated last year
- A curated list of Model Merging methods.☆82Updated last month
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"☆56Updated last month
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (Official Code)☆133Updated last month
- Unofficial implementations of block/layer-wise pruning methods for LLMs.☆49Updated 6 months ago
- Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind☆168Updated 2 months ago
- Language models scale reliably with over-training and on downstream tasks☆94Updated 7 months ago
- ☆73Updated 11 months ago
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆44Updated last year
- Repo for Rho-1: Token-level Data Selection & Selective Pretraining of LLMs.☆302Updated 6 months ago
- ☆42Updated this week