thu-ml / low-bit-optimizersLinks
Low-bit optimizers for PyTorch
☆128Updated last year
Alternatives and similar repositories for low-bit-optimizers
Users that are interested in low-bit-optimizers are comparing it to the libraries listed below
Sorting:
- ☆129Updated 3 months ago
- 🔥 A minimal training framework for scaling FLA models☆146Updated 3 weeks ago
- Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"☆123Updated last year
- Efficient triton implementation of Native Sparse Attention.☆155Updated last week
- Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆161Updated 11 months ago
- [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Training☆203Updated 2 weeks ago
- ☆149Updated last year
- [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…☆101Updated 11 months ago
- ☆53Updated last week
- QuIP quantization☆52Updated last year
- ☆103Updated last year
- ☆74Updated 3 months ago
- ☆222Updated 11 months ago
- Code implementation of GPTQv2 (https://arxiv.org/abs/2504.02692)☆45Updated 2 weeks ago
- ☆45Updated last year
- Flash-Muon: An Efficient Implementation of Muon Optimizer☆115Updated last week
- ☆144Updated 2 years ago
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆163Updated 10 months ago
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆121Updated 4 months ago
- Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks☆143Updated 8 months ago
- [ICLR2025] Codebase for "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing", built on Megatron-LM.☆74Updated 5 months ago
- ☆197Updated 6 months ago
- Official PyTorch implementation of "Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models"☆34Updated last week
- Odysseus: Playground of LLM Sequence Parallelism☆69Updated 11 months ago
- ☆125Updated last year
- Boosting 4-bit inference kernels with 2:4 Sparsity☆75Updated 9 months ago
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆132Updated 11 months ago
- Activation-aware Singular Value Decomposition for Compressing Large Language Models☆67Updated 7 months ago
- [NeurIPS 2024] The official implementation of "Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exitin…☆57Updated 11 months ago
- Unofficial implementations of block/layer-wise pruning methods for LLMs.☆69Updated last year