GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
☆1,712Oct 28, 2024Updated 2 years ago
Alternatives and similar repositories for GaLore
Users that are interested in GaLore are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆207Jul 17, 2024Updated 2 years ago
- Official code for ReLoRA from the paper Stack More Layers Differently: High-Rank Training Through Low-Rank Updates☆477Apr 21, 2024Updated 2 years ago
- Tools for merging pretrained large language models.☆7,436Updated this week
- Efficient Triton Kernels for LLM Training☆6,686Updated this week
- Training LLMs with QLoRA + FSDP☆1,554Nov 9, 2024Updated last year
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793☆462May 13, 2025Updated last year
- [NeurIPS 2024] Low rank memory efficient optimizer without SVD☆34Jul 1, 2025Updated last year
- [NeurIPS 2023] MeZO: Fine-Tuning Language Models with Just Forward Passes. https://arxiv.org/abs/2305.17333☆1,184Jan 11, 2024Updated 2 years ago
- Accessible large language models via k-bit quantization for PyTorch.☆8,536Updated this week
- Official implementation of Half-Quadratic Quantization (HQQ)☆962Feb 26, 2026Updated 8 months ago
- This is the official repository for the paper "Flora: Low-Rank Adapters Are Secretly Gradient Compressors" in ICML 2024.☆106Jul 1, 2024Updated 2 years ago
- Robust recipes to align language models with human and AI preferences☆5,711Sep 23, 2026Updated last month
- PyTorch native post-training library☆5,821Sep 9, 2026Updated last month
- Minimalistic large language model 3D-parallelism training☆2,855Oct 8, 2026Updated 3 weeks ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models(NeurIPS 2024 Spotlight)☆432Jun 30, 2025Updated last year
- Fast and memory-efficient exact attention☆25,249Updated this week
- LOMO: LOw-Memory Optimization☆992Jul 2, 2024Updated 2 years ago
- Schedule-Free Optimization in PyTorch☆2,333Jul 28, 2026Updated 3 months ago
- [ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding☆1,346Mar 6, 2025Updated last year
- Stanford NLP Python library for Representation Finetuning (ReFT)☆1,594Mar 5, 2026Updated 8 months ago
- The official implementation of “Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”☆1,002Jan 30, 2024Updated 2 years ago
- APOLLO: SGD-like Memory, AdamW-level Performance; MLSys'25 Oustanding Paper Honorable Mention☆368Nov 29, 2025Updated 11 months ago
- ☆216Nov 25, 2025Updated 11 months ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)☆2,684Aug 14, 2024Updated 2 years ago
- [ICML 2025] From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories and Applications☆52Oct 30, 2025Updated last year
- [MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration☆3,663Jul 17, 2025Updated last year
- A family of open-sourced Mixture-of-Experts (MoE) Large Language Models☆1,703Mar 8, 2024Updated 2 years ago
- QLoRA: Efficient Finetuning of Quantized LLMs☆11,065Jun 10, 2024Updated 2 years ago
- 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.☆21,871Updated this week
- Reaching LLaMA2 Performance with 0.1M Dollars☆988Jul 23, 2024Updated 2 years ago
- ☆1,600Mar 25, 2026Updated 7 months ago
- A PyTorch native platform for training generative AI models☆5,882Updated this week
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads☆2,773Jun 25, 2024Updated 2 years ago
- [ICLR 2024] Efficient Streaming Language Models with Attention Sinks☆7,275Jul 11, 2024Updated 2 years ago
- 🚀 Efficient implementations for emerging model architectures☆6,019Updated this week
- Go ahead and axolotl questions☆12,669Updated this week
- YaRN: Efficient Context Window Extension of Large Language Models☆1,810Apr 17, 2024Updated 2 years ago
- The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.☆9,008May 3, 2024Updated 2 years ago
- [ICML2024 (Oral)] Official PyTorch implementation of DoRA: Weight-Decomposed Low-Rank Adaptation☆1,006Mar 24, 2026Updated 7 months ago