hundredblocks / large-model-parallelism
Functional local implementations of main model parallelism approaches
☆95Updated 2 years ago
Alternatives and similar repositories for large-model-parallelism
Users that are interested in large-model-parallelism are comparing it to the libraries listed below
Sorting:
- JAX implementation of the Llama 2 model☆218Updated last year
- A puzzle to learn about prompting☆128Updated 2 years ago
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆82Updated last year
- Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*☆82Updated last year
- Resources from the EleutherAI Math Reading Group☆53Updated 2 months ago
- ☆79Updated 10 months ago
- git extension for {collaborative, communal, continual} model development☆213Updated 6 months ago
- An interactive exploration of Transformer programming.☆263Updated last year
- Minimal example scripts of the Hugging Face Trainer, focused on staying under 150 lines☆198Updated last year
- ☆60Updated 3 years ago
- ☆67Updated 2 years ago
- ☆92Updated last year
- Train very large language models in Jax.☆204Updated last year
- some common Huggingface transformers in maximal update parametrization (µP)☆80Updated 3 years ago
- Arrakis is a library to conduct, track and visualize mechanistic interpretability experiments.☆29Updated 3 weeks ago
- Supercharge huggingface transformers with model parallelism.☆76Updated 7 months ago
- Experiments with generating opensource language model assistants☆97Updated 2 years ago
- Training and Inference Notebooks for the RedPajama (OpenLlama) models☆18Updated last year
- Highly commented implementations of Transformers in PyTorch☆136Updated last year
- Inference code for LLaMA models in JAX☆118Updated 11 months ago
- gzip Predicts Data-dependent Scaling Laws☆35Updated 11 months ago
- A library for squeakily cleaning and filtering language datasets.☆47Updated last year
- Python library which enables complex compositions of language models such as scratchpads, chain of thought, tool use, selection-inference…☆207Updated 4 months ago
- Exploring finetuning public checkpoints on filter 8K sequences on Pile☆115Updated 2 years ago
- Automatic gradient descent☆207Updated last year
- QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…☆34Updated last year
- ☆166Updated last year
- HomebrewNLP in JAX flavour for maintable TPU-Training☆50Updated last year
- Proof-of-concept of global switching between numpy/jax/pytorch in a library.☆18Updated 10 months ago
- ☆46Updated 2 years ago