HeegyuKim / torch-xla-SPMDLinks
Pytorch/XLA SPMD Test code in Google TPU
☆23Updated last year
Alternatives and similar repositories for torch-xla-SPMD
Users that are interested in torch-xla-SPMD are comparing it to the libraries listed below
Sorting:
- ☆118Updated last year
- some common Huggingface transformers in maximal update parametrization (µP)☆82Updated 3 years ago
- Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*☆87Updated last year
- Multipack distributed sampler for fast padding-free training of LLMs☆199Updated last year
- Code for Zero-Shot Tokenizer Transfer☆135Updated 7 months ago
- A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.☆70Updated last year
- Understand and test language model architectures on synthetic tasks.☆222Updated last month
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆240Updated 2 months ago
- A fast implementation of T5/UL2 in PyTorch using Flash Attention☆107Updated 5 months ago
- JAX implementation of the Llama 2 model☆219Updated last year
- ☆127Updated last year
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆156Updated 2 months ago
- Explorations into some recent techniques surrounding speculative decoding☆283Updated 8 months ago
- Load compute kernels from the Hub☆258Updated this week
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆262Updated last month
- Simple implementation of Speculative Sampling in NumPy for GPT-2.☆95Updated 2 years ago
- Inference code for LLaMA models in JAX☆118Updated last year
- Simple implementation of muP, based on Spectral Condition for Feature Learning. The implementation is SGD only, dont use it for Adam☆85Updated last year
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆131Updated last year
- Fast, Modern, and Low Precision PyTorch Optimizers☆109Updated last week
- ☆87Updated last year
- supporting pytorch FSDP for optimizers☆84Updated 8 months ago
- Google TPU optimizations for transformers models☆120Updated 7 months ago
- Implementation of the Llama architecture with RLHF + Q-learning☆166Updated 7 months ago
- Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch☆229Updated 11 months ago
- Prune transformer layers☆69Updated last year
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)☆161Updated 4 months ago
- nanoGPT-like codebase for LLM training☆103Updated 3 months ago
- ☆149Updated 2 years ago
- ☆20Updated 2 years ago