Lightning-AI / forked-pdbLinks
Python pdb for multiple processes
☆54Updated 3 months ago
Alternatives and similar repositories for forked-pdb
Users that are interested in forked-pdb are comparing it to the libraries listed below
Sorting:
- Odysseus: Playground of LLM Sequence Parallelism☆76Updated last year
- ☆29Updated 2 years ago
- Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"☆120Updated last year
- ☆106Updated last year
- Triton implementation of FlashAttention2 that adds Custom Masks.☆133Updated last year
- Torch Distributed Experimental☆117Updated last year
- pytorch-profiler☆51Updated 2 years ago
- Fairring (FAIR + Herring) is a plug-in for PyTorch that provides a process group for distributed training that outperforms NCCL at large …☆65Updated 3 years ago
- Megatron's multi-modal data loader☆239Updated last week
- Flash-Muon: An Efficient Implementation of Muon Optimizer☆174Updated 2 months ago
- ☆40Updated last year
- CUDA and Triton implementations of Flash Attention with SoftmaxN.☆73Updated last year
- 🔥 A minimal training framework for scaling FLA models☆233Updated 2 weeks ago
- Training library for Megatron-based models☆38Updated this week
- [EVA ICLR'23; LARA ICML'22] Efficient attention mechanisms via control variates, random features, and importance sampling☆86Updated 2 years ago
- ☆32Updated last year
- Fork of diux-dev/imagenet18☆16Updated 6 years ago
- A performant, memory-efficient checkpointing library for PyTorch applications, designed with large, complex distributed workloads in mind…☆159Updated 2 months ago
- Sparse Backpropagation for Mixture-of-Expert Training☆30Updated last year
- Demystify RAM Usage in Multi-Process Data Loaders☆197Updated 2 years ago
- ☆22Updated 3 months ago
- See details in https://github.com/pytorch/xla/blob/r1.12/torch_xla/distributed/fsdp/README.md☆24Updated 2 years ago
- Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training☆214Updated last year
- Implementation of fused cosine similarity attention in the same style as Flash Attention☆215Updated 2 years ago
- ☆56Updated last year
- 32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.☆49Updated 2 years ago
- ☆149Updated 2 years ago
- This package implements THOR: Transformer with Stochastic Experts.☆65Updated 3 years ago
- 📑 Dive into Big Model Training☆114Updated 2 years ago
- Patch convolution to avoid large GPU memory usage of Conv2D☆92Updated 7 months ago