liangyuwang / zo2
ZO2 (Zeroth-Order Offloading): Full Parameter Fine-Tuning 175B LLMs with 18GB GPU Memory
☆84Updated last week
Alternatives and similar repositories for zo2:
Users that are interested in zo2 are comparing it to the libraries listed below
- Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme☆102Updated last week
- Official code of *Virgo: A Preliminary Exploration on Reproducing o1-like MLLM*☆99Updated last month
- Efficient Mixture of Experts for LLM Paper List☆60Updated 4 months ago
- ☆20Updated last month
- Official code for our paper, "LoRA-Pro: Are Low-Rank Adapters Properly Optimized? "☆111Updated last week
- ☆25Updated 2 months ago
- SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning. COLM 2024 Accepted Paper☆32Updated 10 months ago
- CoT-Valve: Length-Compressible Chain-of-Thought Tuning☆62Updated 2 months ago
- A Framework for Decoupling and Assessing the Capabilities of VLMs☆42Updated 9 months ago
- This project aims to collect and collate various datasets for multimodal large model training, including but not limited to pre-training …☆38Updated 6 months ago
- Official codebase for "GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning".☆41Updated this week
- CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models☆110Updated this week
- The official GitHub page for the survey paper "A Survey on Data Augmentation in Large Model Era"☆124Updated 9 months ago
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆122Updated 3 months ago
- ☆36Updated 7 months ago
- Reformatted Alignment☆115Updated 6 months ago
- [ICLR 2025] MiniPLM: Knowledge Distillation for Pre-Training Language Models☆37Updated 4 months ago
- Parameter-Efficient Fine-Tuning for Foundation Models☆53Updated 2 weeks ago
- ☆91Updated last month
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆131Updated 10 months ago
- ☆73Updated last year
- MMR1: Advancing the Frontiers of Multimodal Reasoning☆153Updated last month
- This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…☆56Updated 11 months ago
- [ACL 2024] The official codebase for the paper "Self-Distillation Bridges Distribution Gap in Language Model Fine-tuning".☆117Updated 5 months ago
- A highly capable 2.4B lightweight LLM using only 1T pre-training data with all details.☆170Updated this week
- Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning☆38Updated 2 months ago
- ☆99Updated 9 months ago
- Open-Pandora: On-the-fly Control Video Generation☆33Updated 4 months ago
- [ICLR 2025] SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction☆66Updated 3 weeks ago
- A generalized framework for subspace tuning methods in parameter efficient fine-tuning.☆136Updated 2 months ago