Alternatives and similar repositories for SOAP:

Users that are interested in SOAP are comparing it to the libraries listed below

• cloneofsimo / zeroshampoo
  ☆33Updated 5 months ago
• evanatyourservice / llm-jax
  Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.
  ☆17Updated 3 weeks ago
• warner-benjamin / optimi
  Fast, Modern, Memory Efficient, and Low Precision PyTorch Optimizers
  ☆84Updated 7 months ago
• amirzandieh / HyperAttention
  Triton Implementation of HyperAttention Algorithm
  ☆47Updated last year
• ethansmith2000 / fsdp_optimizers
  supporting pytorch FSDP for optimizers
  ☆77Updated 2 months ago
• euclaise / supertrainer2000
  ☆49Updated 11 months ago
• crowsonkb / torch-dist-utils
  Utilities for PyTorch distributed
  ☆23Updated this week
• dvruette / barrel-rec-pytorch
  ☆53Updated last year
• ethansmith2000 / TransformerExperiments
  ☆19Updated 5 months ago
• NousResearch / StripedHyenaTrainer
  ☆60Updated last year
• lucaslingle / mu_transformer
  Transformer with Mu-Parameterization, implemented in Jax/Flax. Supports FSDP on TPU pods.
  ☆30Updated 2 months ago
• kaiokendev / cutoff-len-is-context-len
  Demonstration that finetuning RoPE model on larger sequences than the pre-trained model adapts the model context limit
  ☆63Updated last year
• kyleliang919 / Online-Subspace-Descent
  This repo is based on https://github.com/jiaweizzhao/GaLore
  ☆25Updated 5 months ago
• cloneofsimo / min-fsdp
  ☆75Updated 7 months ago
• recursal / GoldFinch-paper
  GoldFinch and other hybrid transformer components
  ☆44Updated 7 months ago
• joey00072 / ohara
  Collection of autoregressive model implementation
  ☆81Updated 2 weeks ago
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆70Updated 4 months ago
• Aleph-Alpha / trigrams
  ☆51Updated 6 months ago
• cloneofsimo / ezmup
  Simple implementation of muP, based on Spectral Condition for Feature Learning. The implementation is SGD only, dont use it for Adam
  ☆73Updated 7 months ago
• epfml / DenseFormer
  ☆78Updated 10 months ago
• cat-state / tinypar
  ☆20Updated last year
• drisspg / transformer_nuggets
  A place to store reusable transformer components of my own creation or found on the interwebs
  ☆47Updated this week
• foundation-model-stack / bamba
  Train, tune, and infer Bamba model
  ☆86Updated last month
• CarperAI / treasure_trove
  ☆22Updated last year
• layer6ai-labs / calo-forest
  A scalable implementation of diffusion and flow-matching with XGBoost models, applied to calorimeter data.
  ☆17Updated 4 months ago
• KhoomeiK / complexity-scaling
  gzip Predicts Data-dependent Scaling Laws
  ☆34Updated 9 months ago
• berlino / seq_icl
  ☆51Updated 9 months ago