tding1 / Efficient-LLM-Survey
The Efficiency Spectrum of LLM
☆52Updated 11 months ago
Related projects ⓘ
Alternatives and complementary repositories for Efficient-LLM-Survey
- Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)☆53Updated last month
- Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)☆79Updated last year
- ☆122Updated 10 months ago
- ☆47Updated 9 months ago
- A Closer Look into Mixture-of-Experts in Large Language Models☆40Updated 3 months ago
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆44Updated last year
- [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts☆34Updated 8 months ago
- Long Context Extension and Generalization in LLMs☆39Updated 2 months ago
- ☆31Updated 2 months ago
- Cascade Speculative Drafting☆26Updated 7 months ago
- Activation-aware Singular Value Decomposition for Compressing Large Language Models☆49Updated last month
- ☆62Updated 3 months ago
- [EMNLP 2023]Context Compression for Auto-regressive Transformers with Sentinel Tokens☆21Updated last year
- From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…☆43Updated 4 months ago
- NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference☆61Updated last month
- Codebase for Instruction Following without Instruction Tuning☆32Updated 2 months ago
- [NeurIPS-2024] 📈 Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies https://arxiv.org/abs/2407.13623☆71Updated last month
- Using FlexAttention to compute attention with different masking patterns☆40Updated 2 months ago
- The implementation for MLSys 2023 paper: "Cuttlefish: Low-rank Model Training without All The Tuning"☆43Updated last year
- This repository contains code for the MicroAdam paper.☆12Updated 4 months ago
- Code for studying the super weight in LLM☆24Updated last week
- Simple and efficient pytorch-native transformer training and inference (batched)☆61Updated 7 months ago
- ☆35Updated 9 months ago
- [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"☆64Updated 5 months ago
- [ICML2024 Spotlight] Fine-Tuning Pre-trained Large Language Models Sparsely☆17Updated 4 months ago
- The official implementation of the paper "Demystifying the Compression of Mixture-of-Experts Through a Unified Framework".☆48Updated last month
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (Official Code)☆135Updated last month
- Pytorch implementation for "Compressed Context Memory For Online Language Model Interaction" (ICLR'24)☆50Updated 7 months ago
- ☆24Updated 8 months ago
- ☆18Updated 3 months ago