gnovack / distributed-training-and-deepspeed
☆17Updated last year
Alternatives and similar repositories for distributed-training-and-deepspeed:
Users that are interested in distributed-training-and-deepspeed are comparing it to the libraries listed below
- Learn CUDA with PyTorch☆19Updated last month
- Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of…☆122Updated 7 months ago
- ☆95Updated 9 months ago
- Various transformers for FSDP research☆37Updated 2 years ago
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆40Updated last year
- A place to store reusable transformer components of my own creation or found on the interwebs☆48Updated last week
- Simple implementation of Speculative Sampling in NumPy for GPT-2.☆92Updated last year
- See https://github.com/cuda-mode/triton-index/ instead!☆11Updated 10 months ago
- Using FlexAttention to compute attention with different masking patterns☆42Updated 6 months ago
- A toolkit for scaling law research ⚖☆48Updated last month
- Experiment of using Tangent to autodiff triton☆78Updated last year
- The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.☆58Updated last month
- ML/DL Math and Method notes☆58Updated last year
- Code for NeurIPS LLM Efficiency Challenge☆57Updated 11 months ago
- Prune transformer layers☆68Updated 9 months ago
- 🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.☆189Updated this week
- ☆76Updated 8 months ago
- ☆19Updated 2 years ago
- Code for studying the super weight in LLM☆91Updated 3 months ago
- Make triton easier☆47Updated 9 months ago
- Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*☆81Updated last year
- Demo of the unit_scaling library, showing how a model can be easily adapted to train in FP8.☆45Updated 8 months ago
- ☆26Updated last year
- Simple and efficient pytorch-native transformer training and inference (batched)☆71Updated 11 months ago
- ☆48Updated last year
- ☆21Updated 2 weeks ago
- NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference☆64Updated 3 months ago
- ☆125Updated last year
- minimal pytorch implementation of bm25 (with sparse tensors)☆97Updated last year