thu-ml / low-bit-optimizersLinks
Low-bit optimizers for PyTorch
☆131Updated last year
Alternatives and similar repositories for low-bit-optimizers
Users that are interested in low-bit-optimizers are comparing it to the libraries listed below
Sorting:
- ☆156Updated 2 years ago
- [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…☆102Updated last year
- ☆129Updated 4 months ago
- Flash-Muon: An Efficient Implementation of Muon Optimizer☆189Updated 3 months ago
- ☆230Updated last year
- ☆142Updated 7 months ago
- Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"☆124Updated last year
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆123Updated 8 months ago
- Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models☆327Updated 7 months ago
- [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Training☆241Updated last month
- Triton implementation of FlashAttention2 that adds Custom Masks.☆135Updated last year
- ☆149Updated 2 years ago
- Activation-aware Singular Value Decomposition for Compressing Large Language Models☆80Updated 11 months ago
- ☆202Updated 10 months ago
- [ICLR2025] Codebase for "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing", built on Megatron-LM.☆95Updated 9 months ago
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆168Updated last year
- Odysseus: Playground of LLM Sequence Parallelism☆77Updated last year
- Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆172Updated last year
- 🔥 A minimal training framework for scaling FLA models☆255Updated 3 weeks ago
- The official implementation of the EMNLP 2023 paper LLM-FP4☆215Updated last year
- Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"☆120Updated last year
- ☆105Updated last year
- Linear Attention Sequence Parallelism (LASP)☆87Updated last year
- Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"☆28Updated last year
- The evaluation framework for training-free sparse attention in LLMs☆100Updated 3 months ago
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆137Updated last year
- Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…☆155Updated 6 months ago
- Reorder-based post-training quantization for large language model☆195Updated 2 years ago
- Triton-based implementation of Sparse Mixture of Experts.☆242Updated this week
- QuIP quantization☆60Updated last year