SparkJiao / llama-pipeline-parallelLinks
A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to copy code and launch discussions about the problems you have encoured.
β56Updated 2 years ago
Alternatives and similar repositories for llama-pipeline-parallel
Users that are interested in llama-pipeline-parallel are comparing it to the libraries listed below
Sorting:
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Modelsβ78Updated last year
- [ICLR 2025] 𧬠RegMix: Data Mixture as Regression for Language Model Pre-training (Spotlight)β157Updated 5 months ago
- [ACL 2024] Long-Context Language Modeling with Parallel Encodingsβ157Updated last year
- β103Updated 3 weeks ago
- [EMNLP 2024] LongAlign: A Recipe for Long Context Alignment of LLMsβ253Updated 7 months ago
- [ACL 2025, Main Conference, Oral] Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Processβ28Updated last year
- Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.β97Updated last year
- [ACL'24] Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningβ166Updated last month
- β107Updated last year
- [ICML 2024] Selecting High-Quality Data for Training Language Modelsβ183Updated last year
- ACL 2024 | LooGLE: Long Context Evaluation for Long-Context Language Modelsβ184Updated 9 months ago
- Homepage for ProLong (Princeton long-context language models) and paper "How to Train Long-Context Language Models (Effectively)"β218Updated 4 months ago
- [ACL-25] We introduce ScaleQuest, a scalable, novel and cost-effective data synthesis method to unleash the reasoning capability of LLMs.β63Updated 9 months ago
- Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scalesβ32Updated 2 years ago
- [NeurIPS 2024] Fast Best-of-N Decoding via Speculative Rejectionβ49Updated 9 months ago
- Counting-Stars (β )β83Updated 2 months ago
- [ACL 2024] MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguesβ102Updated last year
- [ICML'2024] Can AI Assistants Know What They Don't Know?β81Updated last year
- Official repository for ACL 2025 paper "Model Extrapolation Expedites Alignment"β75Updated 2 months ago
- LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelsβ76Updated 9 months ago
- Codes and Data for Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelsβ268Updated 10 months ago
- [ACL 2024 (Oral)] A Prospector of Long-Dependency Data for Large Language Modelsβ56Updated last year
- Unofficial implementation of AlpaGasusβ92Updated last year
- A repository sharing the literatures about long-context large language models, including the methodologies and the evaluation benchmarksβ265Updated last year
- train llama on a single A100 80G node using π€βtransformers and πβDeepspeed Pipeline Parallelismβ224Updated last year
- Code for paper "Patch-Level Training for Large Language Models"β86Updated 8 months ago
- A Comprehensive Survey on Long Context Language Modelingβ169Updated 3 weeks ago
- Towards Systematic Measurement for Long Text Qualityβ37Updated 11 months ago
- Code and Data for "Long-context LLMs Struggle with Long In-context Learning" [TMLR2025]β105Updated 5 months ago
- Rectified Rotary Position Embeddingsβ375Updated last year