Alternatives and similar repositories for ZerO-initialization:

Users that are interested in ZerO-initialization are comparing it to the libraries listed below

• rasbt / cyclemoid-pytorch
  Cyclemoid implementation for PyTorch
  ☆87Updated 2 years ago
• lucidrains / autoregressive-linear-attention-cuda
  CUDA implementation of autoregressive linear attention, with all the latest research findings
  ☆44Updated last year
• lucidrains / compositional-attention-pytorch
  Implementation of "compositional attention" from MILA, a multi-head attention variant that is reframed as a two-step attention process wi…
  ☆50Updated 2 years ago
• FrancescoSaverioZuppichini / Loading-huge-PyTorch-models-with-linear-memory-consumption
  Little article showing how to load pytorch's models with linear memory consumption
  ☆34Updated 2 years ago
• lucidrains / hourglass-transformer-pytorch
  Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI
  ☆84Updated 3 years ago
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆112Updated 3 months ago
• lucidrains / kalman-filtering-attention
  Implementation of the Kalman Filtering Attention proposed in "Kalman Filtering Attention for User Behavior Modeling in CTR Prediction"
  ☆57Updated last year
• ahennequ / pytorch-custom-mma
  ☆29Updated 2 years ago
• peerdavid / layerwise-batch-entropy
  Layerwise Batch Entropy Regularization
  ☆22Updated 2 years ago
• lucidrains / taylor-series-linear-attention
  Explorations into the recently proposed Taylor Series Linear Attention
  ☆91Updated 4 months ago
• andravin / spio
  Efficient CUDA kernels for training convolutional neural networks with PyTorch.
  ☆38Updated last month
• lucidrains / gateloop-transformer
  Implementation of GateLoop Transformer in Pytorch and Jax
  ☆87Updated 6 months ago
• ClashLuke / tpucare
  Automatically take good care of your preemptible TPUs
  ☆34Updated last year
• lucidrains / gated-state-spaces-pytorch
  Implementation of Gated State Spaces, from the paper "Long Range Language Modeling via Gated State Spaces", in Pytorch
  ☆97Updated last year
• lucidrains / panoptic-transformer
  Another attempt at a long-context / efficient transformer by me
  ☆37Updated 2 years ago
• HomebrewNLP / HomebrewNLP
  A case study of efficient training of large language models using commodity hardware.
  ☆68Updated 2 years ago
• sayakpaul / MLP-Mixer-CIFAR10
  Implements MLP-Mixer (https://arxiv.org/abs/2105.01601) with the CIFAR-10 dataset.
  ☆54Updated 2 years ago
• HomebrewNLP / Olmax
  HomebrewNLP in JAX flavour for maintable TPU-Training
  ☆47Updated 11 months ago
• gregorbachmann / scaling_mlps
  ☆51Updated 7 months ago
• lucidrains / Adan-pytorch
  Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch
  ☆250Updated 2 years ago
• lucidrains / einops-exts
  Implementation of some personal helper functions for Einops, my most favorite tensor manipulation library ❤️
  ☆53Updated 2 years ago
• kshitij12345 / torchnnprofiler
  Context Manager to profile the forward and backward times of PyTorch's nn.Module
  ☆84Updated last year
• Ujjawal-K-Panchal / coshnet
  ☆48Updated 9 months ago
• lessw2020 / transformer_central
  Various transformers for FSDP research
  ☆34Updated 2 years ago
• matthias-wright / jax-fid
  FID computation in Jax/Flax.
  ☆26Updated 6 months ago
• sayakpaul / big_vision_experiments
  Contains my experiments with the `big_vision` repo to train ViTs on ImageNet-1k.
  ☆22Updated 2 years ago
• lucidrains / logavgexp-torch
  Implementation of LogAvgExp for Pytorch
  ☆32Updated 2 years ago
• lucidrains / discrete-key-value-bottleneck-pytorch
  Implementation of Discrete Key / Value Bottleneck, in Pytorch
  ☆87Updated last year