SparkJiao / llama-pipeline-parallelLinks
A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to copy code and launch discussions about the problems you have encoured.
β56Updated 2 years ago
Alternatives and similar repositories for llama-pipeline-parallel
Users that are interested in llama-pipeline-parallel are comparing it to the libraries listed below
Sorting:
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Modelsβ78Updated last year
- [ICLR 2025] 𧬠RegMix: Data Mixture as Regression for Language Model Pre-training (Spotlight)β164Updated 6 months ago
- β104Updated last month
- [ACL 2024] Long-Context Language Modeling with Parallel Encodingsβ157Updated last year
- [EMNLP 2024] LongAlign: A Recipe for Long Context Alignment of LLMsβ253Updated 8 months ago
- Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.β98Updated last year
- [ACL 2025, Main Conference, Oral] Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Processβ30Updated last year
- [ICML 2024] Selecting High-Quality Data for Training Language Modelsβ185Updated last year
- β109Updated last year
- [ACL-25] We introduce ScaleQuest, a scalable, novel and cost-effective data synthesis method to unleash the reasoning capability of LLMs.β64Updated 10 months ago
- Homepage for ProLong (Princeton long-context language models) and paper "How to Train Long-Context Language Models (Effectively)"β221Updated 5 months ago
- [ACL'24] Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningβ170Updated 2 months ago
- Implementation of NAACL 2024 Outstanding Paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"β148Updated 5 months ago
- ACL 2024 | LooGLE: Long Context Evaluation for Long-Context Language Modelsβ185Updated 10 months ago
- Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scalesβ32Updated 2 years ago
- Counting-Stars (β )β83Updated 2 months ago
- [NeurIPS 2024] Fast Best-of-N Decoding via Speculative Rejectionβ51Updated 9 months ago
- [ICML'24] The official implementation of βRethinking Optimization and Architecture for Tiny Language Modelsββ123Updated 7 months ago
- [ACL 2024 (Oral)] A Prospector of Long-Dependency Data for Large Language Modelsβ56Updated last year
- LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelsβ76Updated 10 months ago
- Codes and Data for Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelsβ267Updated 11 months ago
- β18Updated 8 months ago
- Code for paper "Patch-Level Training for Large Language Models"β86Updated 9 months ago
- Towards Systematic Measurement for Long Text Qualityβ37Updated 11 months ago
- [ICML'2024] Can AI Assistants Know What They Don't Know?β83Updated last year
- Unofficial implementation of AlpaGasusβ92Updated last year
- Repository of LV-Eval Benchmarkβ69Updated 11 months ago
- [ACL 2024] MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguesβ105Updated last year
- β98Updated 4 months ago
- code for Scaling Laws of RoPE-based Extrapolationβ73Updated last year