gnovack / distributed-training-and-deepspeedLinks
☆17Updated 2 years ago
Alternatives and similar repositories for distributed-training-and-deepspeed
Users that are interested in distributed-training-and-deepspeed are comparing it to the libraries listed below
Sorting:
- Various transformers for FSDP research☆37Updated 2 years ago
- A place to store reusable transformer components of my own creation or found on the interwebs☆56Updated last week
- Context Manager to profile the forward and backward times of PyTorch's nn.Module☆83Updated last year
- ☆28Updated 6 months ago
- ML/DL Math and Method notes☆61Updated last year
- ☆19Updated 2 years ago
- This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…☆92Updated 2 years ago
- CUDA and Triton implementations of Flash Attention with SoftmaxN.☆70Updated last year
- ☆79Updated last year
- Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of…☆138Updated 11 months ago
- experiments with inference on llama☆104Updated last year
- 🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.☆205Updated last week
- Make triton easier☆47Updated last year
- Experiment of using Tangent to autodiff triton☆79Updated last year
- Simple implementation of Speculative Sampling in NumPy for GPT-2.☆95Updated last year
- ☆21Updated 4 months ago
- Code for NeurIPS LLM Efficiency Challenge☆59Updated last year
- ☆18Updated last week
- Supercharge huggingface transformers with model parallelism.☆77Updated 9 months ago
- Small scale distributed training of sequential deep learning models, built on Numpy and MPI.☆134Updated last year
- Simple and efficient pytorch-native transformer training and inference (batched)☆77Updated last year
- Learn CUDA with PyTorch☆29Updated this week
- See https://github.com/cuda-mode/triton-index/ instead!☆11Updated last year
- a pipeline for using api calls to agnostically convert unstructured data into structured training data☆30Updated 9 months ago
- ☆112Updated last year
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆59Updated 9 months ago
- A library for squeakily cleaning and filtering language datasets.☆47Updated 2 years ago
- Torch Distributed Experimental☆116Updated 11 months ago
- Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*☆85Updated last year
- Pragmatic approach to parsing import profiles for CI's☆11Updated last year