rasbt / pytorch-memory-optimLinks
This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog post.
☆92Updated 2 years ago
Alternatives and similar repositories for pytorch-memory-optim
Users that are interested in pytorch-memory-optim are comparing it to the libraries listed below
Sorting:
- ML/DL Math and Method notes☆63Updated last year
- Collection of autoregressive model implementation☆86Updated 4 months ago
- Implementation of the Llama architecture with RLHF + Q-learning☆166Updated 6 months ago
- A place to store reusable transformer components of my own creation or found on the interwebs☆60Updated 2 weeks ago
- Various transformers for FSDP research☆38Updated 2 years ago
- experiments with inference on llama☆104Updated last year
- Code for NeurIPS LLM Efficiency Challenge☆59Updated last year
- ☆87Updated last year
- Just some miscellaneous utility functions / decorators / modules related to Pytorch and Accelerate to help speed up implementation of new…☆124Updated last year
- ☆163Updated last year
- ☆88Updated last year
- LoRA and DoRA from Scratch Implementations☆209Updated last year
- 👷 Build compute kernels☆106Updated last week
- Experiment of using Tangent to autodiff triton☆80Updated last year
- CUDA and Triton implementations of Flash Attention with SoftmaxN.☆73Updated last year
- ☆88Updated last year
- ☆48Updated 11 months ago
- A really tiny autograd engine☆95Updated 3 months ago
- Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*☆87Updated last year
- An introduction to LLM Sampling☆79Updated 8 months ago
- ☆82Updated last year
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆131Updated last year
- Utilities for Training Very Large Models☆58Updated 11 months ago
- LLM training in simple, raw C/CUDA☆15Updated 8 months ago
- Triton Implementation of HyperAttention Algorithm☆48Updated last year
- Context Manager to profile the forward and backward times of PyTorch's nn.Module☆83Updated last year
- Demo of the unit_scaling library, showing how a model can be easily adapted to train in FP8.☆46Updated last year
- ☆23Updated 2 years ago
- Repository containing awesome resources regarding Hugging Face tooling.☆48Updated last year
- Genalog is an open source, cross-platform python package allowing generation of synthetic document images with custom degradations and te…☆42Updated last year