hundredblocks / large-model-parallelismLinks
Functional local implementations of main model parallelism approaches
☆95Updated 2 years ago
Alternatives and similar repositories for large-model-parallelism
Users that are interested in large-model-parallelism are comparing it to the libraries listed below
Sorting:
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆81Updated last year
- Inference code for LLaMA models in JAX☆117Updated last year
- Resources from the EleutherAI Math Reading Group☆53Updated 3 months ago
- ☆78Updated 11 months ago
- Train very large language models in Jax.☆204Updated last year
- JAX implementation of the Llama 2 model☆217Updated last year
- Automatic gradient descent☆207Updated last year
- Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*☆82Updated last year
- A puzzle to learn about prompting☆127Updated 2 years ago
- gzip Predicts Data-dependent Scaling Laws☆35Updated last year
- ☆22Updated last year
- ☆49Updated last year
- An interactive exploration of Transformer programming.☆264Updated last year
- Solve puzzles. Learn CUDA.☆64Updated last year
- Code associated to papers on superposition (in ML interpretability)☆28Updated 2 years ago
- Highly commented implementations of Transformers in PyTorch☆136Updated last year
- Proof-of-concept of global switching between numpy/jax/pytorch in a library.☆18Updated 11 months ago
- ☆60Updated last year
- Automatically take good care of your preemptible TPUs☆36Updated 2 years ago
- Minimal example scripts of the Hugging Face Trainer, focused on staying under 150 lines☆197Updated last year
- HomebrewNLP in JAX flavour for maintable TPU-Training☆50Updated last year
- ☆53Updated last year
- A really tiny autograd engine☆94Updated last week
- Exploring finetuning public checkpoints on filter 8K sequences on Pile☆114Updated 2 years ago
- ☆92Updated last year
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆127Updated last year
- ☆143Updated 2 years ago
- Experiments with generating opensource language model assistants☆97Updated 2 years ago
- nanoGPT-like codebase for LLM training☆94Updated 2 weeks ago
- some common Huggingface transformers in maximal update parametrization (µP)☆80Updated 3 years ago