aitsc / GLMKD
Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method ; GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
β32Updated last year
Alternatives and similar repositories for GLMKD:
Users that are interested in GLMKD are comparing it to the libraries listed below
- β98Updated 6 months ago
- [ICLR 2025] 𧬠RegMix: Data Mixture as Regression for Language Model Pre-training (Spotlight)β117Updated last month
- [ICML 2024] Selecting High-Quality Data for Training Language Modelsβ158Updated 9 months ago
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Modelsβ76Updated last year
- [ICLR'24 spotlight] Tool-Augmented Reward Modelingβ46Updated 3 months ago
- Code for preprint "Metadata Conditioning Accelerates Language Model Pre-training (MeCo)"β36Updated last week
- An Experiment on Dynamic NTK Scaling RoPEβ62Updated last year
- [ACL 2024] Long-Context Language Modeling with Parallel Encodingsβ153Updated 9 months ago
- Implementations of online merging optimizers proposed by Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentβ74Updated 9 months ago
- β64Updated 11 months ago
- [ACL'24] Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningβ147Updated 6 months ago
- β48Updated 11 months ago
- Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Processβ24Updated 7 months ago
- Towards Systematic Measurement for Long Text Qualityβ34Updated 6 months ago
- Code for M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Modelsβ22Updated 8 months ago
- A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free toβ¦β55Updated last year
- Code implementation of synthetic continued pretrainingβ95Updated 2 months ago
- [ICLR 2025] MiniPLM: Knowledge Distillation for Pre-Training Language Modelsβ34Updated 4 months ago
- Code for ACL2023 paper: Pre-Training to Learn in Contextβ108Updated 8 months ago
- Official repository for MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models [NeurIPS 2024]β61Updated 4 months ago
- The code of paper "Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation" published at NeurIPS 202β¦β45Updated 2 years ago
- We introduce ScaleQuest, a scalable, novel and cost-effective data synthesis method to unleash the reasoning capability of LLMs.β60Updated 5 months ago
- [NeurIPS 2023] Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectiveβ30Updated last year
- [NeurIPS 2024 Main Track] Code for the paper titled "Instruction Tuning With Loss Over Instructions"β35Updated 10 months ago
- One Network, Many Masks: Towards More Parameter-Efficient Transfer Learningβ39Updated last year
- Official code for "MAmmoTH2: Scaling Instructions from the Web" [NeurIPS 2024]β137Updated 5 months ago
- Repository for "Propagating Knowledge Updates to LMs Through Distillation" (NeurIPS 2023).β25Updated 7 months ago
- β45Updated 9 months ago
- The code and data for the paper JiuZhang3.0β43Updated 10 months ago
- β30Updated 6 months ago