Alternatives and similar repositories for ZeRO-transformer:

Users that are interested in ZeRO-transformer are comparing it to the libraries listed below

• young-geng / mintext
  Minimal but scalable implementation of large language models in JAX
  ☆34Updated 4 months ago
• MatX-inc / seqax
  seqax = sequence modeling + JAX
  ☆150Updated last week
• young-geng / scalax
  A simple library for scaling up JAX programs
  ☆134Updated 4 months ago
• davisyoshida / lorax
  LoRA for arbitrary JAX models and functions
  ☆135Updated last year
• sholtodouglas / multihost_dataloading
  Experimenting with how best to do multi-host dataloading
  ☆10Updated 2 years ago
• jax-ml / jax-llm-examples
  ☆87Updated last week
• Sea-Snell / JAX_llama
  Inference code for LLaMA models in JAX
  ☆116Updated 10 months ago
• google-deepmind / nanodo
  ☆214Updated 8 months ago
• lucaslingle / mu_transformer
  Transformer with Mu-Parameterization, implemented in Jax/Flax. Supports FSDP on TPU pods.
  ☆30Updated 3 months ago
• davisyoshida / qax
  If it quacks like a tensor...
  ☆57Updated 4 months ago
• srush / triton-autodiff
  Experiment of using Tangent to autodiff triton
  ☆78Updated last year
• cloneofsimo / min-fsdp
  ☆76Updated 8 months ago
• nshepperd / flash_attn_jax
  JAX bindings for Flash Attention v2
  ☆88Updated 8 months ago
• young-geng / mlxu
  Machine Learning eXperiment Utilities
  ☆46Updated 9 months ago
• shawntan / scattermoe
  Triton-based implementation of Sparse Mixture of Experts.
  ☆208Updated 3 months ago
• sholtodouglas / scalingExperiments
  ☆58Updated 3 years ago
• google / praxis
  ☆184Updated last month
• AllanYangZhou / midGPT
  Distributed pretraining of large language models (LLMs) on cloud TPU slices, with Jax and Equinox.
  ☆24Updated 5 months ago
• graphcore-research / unit-scaling
  A library for unit scaling in PyTorch
  ☆124Updated 3 months ago
• vdesai2014 / inference-optimization-blog-post
  ☆86Updated last year
• alvarobartt / safejax
  Serialize JAX, Flax, Haiku, or Objax model params with 🤗`safetensors`
  ☆44Updated 9 months ago
• stanford-crfm / haliax
  Named Tensors for Legible Deep Learning in JAX
  ☆167Updated this week
• NX-AI / flashrnn
  FlashRNN - Fast RNN Kernels with I/O Awareness
  ☆76Updated last week
• cloneofsimo / min-max-gpt
  Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training
  ☆122Updated 11 months ago
• graphcore-research / out-of-the-box-fp8-training
  Demo of the unit_scaling library, showing how a model can be easily adapted to train in FP8.
  ☆45Updated 8 months ago
• Edward-Sun / gpt-accelera
  Simple and efficient pytorch-native transformer training and inference (batched)
  ☆71Updated 11 months ago
• cchan / tccl
  extensible collectives library in triton
  ☆84Updated 6 months ago
• jenkspt / gpt-jax
  Jax/Flax rewrite of Karpathy's nanoGPT
  ☆57Updated 2 years ago
• pytorch-labs / float8_experimental
  This repository contains the experimental PyTorch native float8 training UX
  ☆222Updated 7 months ago
• lianakoleva / no-libtorch-compile
  ☆21Updated 3 weeks ago