A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to copy code and launch discussions about the problems you have encoured.
☆57Jul 4, 2023Updated 3 years ago
Alternatives and similar repositories for llama-pipeline-parallel
Users that are interested in llama-pipeline-parallel are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.☆98Feb 5, 2024Updated 2 years ago
- train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism☆224Nov 21, 2023Updated 2 years ago
- [EMNLP 2024] Source code for the paper "Learning Planning-based Reasoning with Trajectory Collection and Process Rewards Synthesizing".☆84Jan 14, 2025Updated last year
- ☆19Jul 24, 2025Updated last year
- Deferred Continuous Batching in Resource-Efficient Large Language Model Serving (EuroMLSys 2024)☆20May 28, 2024Updated 2 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- [ICLR 2022] Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators☆27Jul 26, 2023Updated 3 years ago
- ☆27Aug 25, 2023Updated 3 years ago
- ☆15Dec 28, 2022Updated 3 years ago
- [COLING2022] Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection☆10Aug 3, 2023Updated 3 years ago
- code for COLING paper "A Hybrid Model of Classification and Generation for Spatial Relation Extraction"☆10Oct 20, 2022Updated 4 years ago
- LLM Training code from our paper **Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Gener…☆14Aug 7, 2026Updated 2 months ago
- Mosaic Representation Learning for Self-supervised Visual Pre-training (ICLR2023, Spotlight)☆15Apr 7, 2023Updated 3 years ago
- ☆12May 11, 2025Updated last year
- ICML'25: The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models☆11May 19, 2025Updated last year
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Customized Inference Engine for Multiverse Models☆26Jun 27, 2025Updated last year
- ☆11Oct 8, 2023Updated 3 years ago
- ☆17Oct 15, 2023Updated 3 years ago
- LP_Bench☆18Feb 27, 2026Updated 8 months ago
- ☆11Aug 15, 2023Updated 3 years ago
- Official implementation of the paper "From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large L…☆55Jun 24, 2024Updated 2 years ago
- simple implementation of Expected Gradients and Integrated Gradients by pytorch☆13May 11, 2022Updated 4 years ago
- [ICML 2026] Improving GPT via a simple normalization strategy☆15May 22, 2026Updated 5 months ago
- ☆12Apr 19, 2024Updated 2 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- 集中管理所有的prompt。☆14Nov 27, 2024Updated last year
- Code and data for the paper ”Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a L…☆10Dec 3, 2023Updated 2 years ago
- An open-source library for contamination detection in NLP datasets and Large Language Models (LLMs).☆60Aug 13, 2024Updated 2 years ago
- ☆29Dec 2, 2024Updated last year
- The appendix and core code of model CauSTG, for accepted paper in KDD 2023.☆12Jun 15, 2023Updated 3 years ago
- Repo for the EMNLP2021 paper: Lifelong Event Detection with Knowledge Transfer☆14Sep 2, 2021Updated 5 years ago
- This code is a official PyTorch implementation of our paper "Low-rank Adaptation for Spatio-Temporal Forecasting".☆12Sep 30, 2025Updated last year
- ☆14Jul 13, 2022Updated 4 years ago
- Repository for paper ``IG2: Integrated Gradient on Iterative Gradient Path for Feature Attribution‘’☆14Feb 28, 2025Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Distributed SDDMM Kernel☆13Jul 8, 2022Updated 4 years ago
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆762Sep 27, 2024Updated 2 years ago
- SHADE: Enable Fundamental Cacheability for Distributed Deep Learning Training☆36Mar 1, 2023Updated 3 years ago
- Source code for ICLR 2021 paper : Pre-training Text-to-Text Transformers for Concept-Centric Common Sense☆25Sep 16, 2021Updated 5 years ago
- A benchmark to evaluate search-augmented LLMs☆17Aug 28, 2025Updated last year
- [EMNLP 2022] Language Model Pre-Training with Sparse Latent Typing☆14Feb 10, 2023Updated 3 years ago
- Mamba SSM architecture for packed data☆10Jul 1, 2024Updated 2 years ago