thu-ml / low-bit-optimizers
Low-bit optimizers for PyTorch
β125Updated last year
Alternatives and similar repositories for low-bit-optimizers:
Users that are interested in low-bit-optimizers are comparing it to the libraries listed below
- β145Updated last year
- π₯ A minimal training framework for scaling FLA modelsβ92Updated last week
- [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear atβ¦β99Updated 9 months ago
- [ICML'24] The official implementation of βRethinking Optimization and Architecture for Tiny Language Modelsββ121Updated 2 months ago
- [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Trainingβ170Updated last week
- β122Updated last month
- β102Updated last year
- Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"β123Updated 11 months ago
- β220Updated 9 months ago
- The official implementation of the EMNLP 2023 paper LLM-FP4β192Updated last year
- Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformersβ207Updated 7 months ago
- Efficient triton implementation of Native Sparse Attention.β127Updated this week
- Odysseus: Playground of LLM Sequence Parallelismβ68Updated 9 months ago
- Reorder-based post-training quantization for large language modelβ186Updated last year
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelsβ130Updated 9 months ago
- Activation-aware Singular Value Decomposition for Compressing Large Language Modelsβ60Updated 5 months ago
- Boosting 4-bit inference kernels with 2:4 Sparsityβ72Updated 6 months ago
- Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMsβ80Updated 4 months ago
- XAttention: Block Sparse Attention with Antidiagonal Scoringβ118Updated this week
- [ICLR2025] Codebase for "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing", built on Megatron-LM.β66Updated 3 months ago
- An algorithm for weight-activation quantization (W4A4, W4A8) of LLMs, supporting both static and dynamic quantizationβ124Updated 2 months ago
- PB-LLM: Partially Binarized Large Language Modelsβ152Updated last year
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLMβ158Updated 8 months ago
- Linear Attention Sequence Parallelism (LASP)β81Updated 9 months ago
- Triton-based implementation of Sparse Mixture of Experts.β210Updated 4 months ago
- β125Updated last year
- β141Updated last year
- β185Updated 6 months ago
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)β90Updated 2 weeks ago
- [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"β77Updated 9 months ago