rasbt / pytorch-memory-optimLinks
This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog post.
☆92Updated 2 years ago
Alternatives and similar repositories for pytorch-memory-optim
Users that are interested in pytorch-memory-optim are comparing it to the libraries listed below
Sorting:
- Collection of autoregressive model implementation☆86Updated 5 months ago
- Various transformers for FSDP research☆38Updated 2 years ago
- Implementation of the Llama architecture with RLHF + Q-learning☆168Updated 8 months ago
- A place to store reusable transformer components of my own creation or found on the interwebs☆60Updated 2 weeks ago
- ML/DL Math and Method notes☆63Updated last year
- ☆89Updated last year
- Experiment of using Tangent to autodiff triton☆81Updated last year
- Triton Implementation of HyperAttention Algorithm☆48Updated last year
- ☆48Updated last year
- CUDA and Triton implementations of Flash Attention with SoftmaxN.☆73Updated last year
- ☆173Updated last year
- Code for NeurIPS LLM Efficiency Challenge☆59Updated last year
- LoRA and DoRA from Scratch Implementations☆211Updated last year
- ☆82Updated last year
- Demo of the unit_scaling library, showing how a model can be easily adapted to train in FP8.☆46Updated last year
- ☆89Updated last year
- Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*☆87Updated last year
- PB-LLM: Partially Binarized Large Language Models☆155Updated last year
- ☆69Updated last year
- Utilities for Training Very Large Models☆58Updated last year
- ☆88Updated last year
- Simple implementation of Speculative Sampling in NumPy for GPT-2.☆96Updated 2 years ago
- Implements Low-Rank Adaptation(LoRA) Finetuning from scratch☆80Updated 2 years ago
- experiments with inference on llama☆104Updated last year
- Simple and efficient pytorch-native transformer training and inference (batched)☆79Updated last year
- ☆38Updated last year
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆132Updated last year
- NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day☆256Updated last year
- Just some miscellaneous utility functions / decorators / modules related to Pytorch and Accelerate to help speed up implementation of new…☆125Updated last year
- Token Omission Via Attention☆128Updated 11 months ago