A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to copy code and launch discussions about the problems you have encoured.
☆58Jul 4, 2023Updated 2 years ago
Alternatives and similar repositories for llama-pipeline-parallel
Users that are interested in llama-pipeline-parallel are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.☆97Feb 5, 2024Updated 2 years ago
- train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism☆223Nov 21, 2023Updated 2 years ago
- LLM KV Cache compression - K+V dual compression, 73-99% VRAM savings, zero accuracy loss☆49Mar 30, 2026Updated 2 weeks ago
- [EMNLP 2024] Source code for the paper "Learning Planning-based Reasoning with Trajectory Collection and Process Rewards Synthesizing".☆83Jan 14, 2025Updated last year
- ☆19Jul 24, 2025Updated 8 months ago
- Deploy open-source AI quickly and easily - Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- [ICLR 2022] Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators☆26Jul 26, 2023Updated 2 years ago
- Code for ACL 2023 Oral Paper: ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning☆12Aug 23, 2025Updated 7 months ago
- Slowdown prediction module of Echo: Simulating Distributed Training at Scale☆13May 17, 2025Updated 11 months ago
- Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning☆20Feb 4, 2022Updated 4 years ago
- Official implementation of the paper "From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large L…☆53Jun 24, 2024Updated last year
- Mosaic Representation Learning for Self-supervised Visual Pre-training (ICLR2023, Spotlight)☆15Apr 7, 2023Updated 3 years ago
- Fully open reproduction of DeepSeek-R1☆11Mar 24, 2025Updated last year
- Unsupervised Cross-lingual Sentiment Analysis (CoNLL 2019)☆10Nov 4, 2019Updated 6 years ago
- 集中管理所有的prompt。☆14Nov 27, 2024Updated last year
- Wordpress hosting with auto-scaling - Free Trial • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- a source code for automatic data visualization and recommendation☆14Jul 12, 2018Updated 7 years ago
- An open-source library for contamination detection in NLP datasets and Large Language Models (LLMs).☆61Aug 13, 2024Updated last year
- ☆11Aug 15, 2023Updated 2 years ago
- Official Codebase for "Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control" (NeurIPS 2024)☆15Oct 29, 2024Updated last year
- GEMV implementation with CUTLASS☆21Aug 21, 2025Updated 7 months ago
- ☆17Oct 15, 2023Updated 2 years ago
- Pipeline Parallelism for PyTorch☆786Aug 21, 2024Updated last year
- Distributed SDDMM Kernel☆12Jul 8, 2022Updated 3 years ago
- ☆14Jul 13, 2022Updated 3 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆755Sep 27, 2024Updated last year
- 🖖 图谱式笔记系统，旨在提高个人笔记的使用率！☆12Jan 17, 2021Updated 5 years ago
- [ACL 2021 Findings] HySPA: Hybrid Span Generation for Scalable Text-to-Graph Extraction☆10Sep 16, 2021Updated 4 years ago
- ☆15May 15, 2025Updated 11 months ago
- [EMNLP 2022] Language Model Pre-Training with Sparse Latent Typing☆14Feb 10, 2023Updated 3 years ago
- ☆16Apr 11, 2022Updated 4 years ago
- Google DeepMind: Mixture of Depths Unofficial Implementation.☆12May 29, 2024Updated last year
- [TBD] "m4: A Learned Flow-level Network Simulator" by Chenning Li, Anton A. Zabreyko, Om Chabra, Arash Nasr-Esfahany, Kevin Zhao, Pratees…☆18Nov 18, 2025Updated 5 months ago
- [3DV 2025] CoE: Deep Coupled Embedding for Non-Rigid Point Cloud Correspondences☆19Jan 5, 2026Updated 3 months ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Cursor IDE (v2.6.22) backend endpoint API reverse engineered☆54Apr 2, 2026Updated 2 weeks ago
- Source code for ICLR 2021 paper : Pre-training Text-to-Text Transformers for Concept-Centric Common Sense☆26Sep 16, 2021Updated 4 years ago
- 高性能文本 Tokenizer 库☆31Feb 2, 2024Updated 2 years ago
- Code for Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach☆14Jul 19, 2020Updated 5 years ago
- The Web Conference 2020: Structure-Feature based Graph Self-adaptive Pooling☆30Apr 21, 2020Updated 5 years ago
- Ring attention implementation with flash attention☆1,006Sep 10, 2025Updated 7 months ago
- Code for the paper "Query-Key Normalization for Transformers"☆52Mar 6, 2021Updated 5 years ago