hwang595 / Cuttlefish
The implementation for MLSys 2023 paper: "Cuttlefish: Low-rank Model Training without All The Tuning"
☆44Updated last year
Alternatives and similar repositories for Cuttlefish:
Users that are interested in Cuttlefish are comparing it to the libraries listed below
- ☆50Updated last year
- Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)☆80Updated last year
- Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"☆27Updated last year
- ☆69Updated 2 months ago
- Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"☆116Updated last year
- ☆36Updated 7 months ago
- ☆31Updated last year
- ☆81Updated last year
- 32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.☆47Updated last year
- [ICML2024 Spotlight] Fine-Tuning Pre-trained Large Language Models Sparsely☆23Updated 10 months ago
- [ICLR 2025] Official PyTorch implementation of "Forgetting Transformer: Softmax Attention with a Forget Gate"☆95Updated 2 weeks ago
- [ICML‘2024] "LoCoCo: Dropping In Convolutions for Long Context Compression", Ruisi Cai, Yuandong Tian, Zhangyang Wang, Beidi Chen☆16Updated 7 months ago
- Here we will test various linear attention designs.☆60Updated last year
- Stick-breaking attention☆52Updated last month
- Official code for the paper "Attention as a Hypernetwork"☆30Updated 10 months ago
- Linear Attention Sequence Parallelism (LASP)☆82Updated 10 months ago
- [ACL 2024] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models☆87Updated 11 months ago
- Beyond KV Caching: Shared Attention for Efficient LLMs☆18Updated 9 months ago
- Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMs☆81Updated 5 months ago
- ☆102Updated last year
- Sparse Backpropagation for Mixture-of-Expert Training☆29Updated 9 months ago
- Activation-aware Singular Value Decomposition for Compressing Large Language Models☆63Updated 6 months ago
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆51Updated 2 years ago
- Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)☆57Updated 6 months ago
- ☆16Updated last year
- From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…☆44Updated this week
- ☆122Updated 2 months ago
- Low-bit optimizers for PyTorch☆128Updated last year
- A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.☆65Updated 8 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆59Updated 6 months ago