Alternatives and similar repositories for grokfast:

Users that are interested in grokfast are comparing it to the libraries listed below

• srush / annotated-mamba
  Annotated version of the Mamba paper
  ☆481Updated last year
• lucidrains / nGPT-pytorch
  Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAI
  ☆279Updated last month
• bloc97 / DeMo
  DeMo: Decoupled Momentum Optimization
  ☆186Updated 4 months ago
• microsoft / Samba
  [ICLR 2025] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
  ☆862Updated 2 months ago
• NVIDIA / ngpt
  Normalized Transformer (nGPT)
  ☆168Updated 5 months ago
• HomebrewML / HeavyBall
  Efficient optimizers
  ☆189Updated this week
• KellerJordan / Muon
  Muon optimizer: +>30% sample efficiency with <3% wallclock overhead
  ☆575Updated 3 weeks ago
• facebookresearch / optimizers
  For optimization algorithm research and development.
  ☆507Updated this week
• KindXiaoming / grow-crystals
  Getting crystal-like representations with harmonic loss
  ☆182Updated 2 weeks ago
• PeaBrane / mamba-tiny
  Simple, minimal implementation of the Mamba SSM in one pytorch file. Using logcumsumexp (Heisen sequence).
  ☆112Updated 6 months ago
• LucasPrietoAl / grokking-at-the-edge-of-numerical-stability
  ☆93Updated 3 months ago
• nanowell / AdEMAMix-Optimizer-Pytorch
  The AdEMAMix Optimizer: Better, Faster, Older.
  ☆180Updated 7 months ago
• modula-systems / modula
  🧱 Modula software package
  ☆188Updated 3 weeks ago
• Haiyang-W / TokenFormer
  [ICLR2025 Spotlight🔥] Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters
  ☆548Updated 2 months ago
• kvfrans / jax-diffusion-transformer
  Implementation of Diffusion Transformer (DiT) in JAX
  ☆270Updated 10 months ago
• nikhilvyas / SOAP
  ☆173Updated 4 months ago
• srush / annotated-s4
  Implementation of https://srush.github.io/annotated-s4
  ☆489Updated 2 years ago
• CG80499 / KAN-GPT-2
  Training small GPT-2 style models using Kolmogorov-Arnold networks.
  ☆116Updated 10 months ago
• zyushun / Adam-mini
  Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793
  ☆405Updated this week
• iShohei220 / adopt
  Official Implementation of "ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Rate"
  ☆424Updated 4 months ago
• SakanaAI / evo-memory
  Code to train and evaluate Neural Attention Memory Models to obtain universally-applicable memory systems for transformers.
  ☆304Updated 5 months ago
• lucidrains / ring-attention-pytorch
  Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
  ☆510Updated 5 months ago
• jerber / lang-jepa
  ☆108Updated 3 months ago
• stanford-crfm / levanter
  Legible, Scalable, Reproducible Foundation Models with Named Tensors and Jax
  ☆566Updated this week
• google-deepmind / nanodo
  ☆215Updated 9 months ago
• google-deepmind / recurrentgemma
  Open weights language model from Google DeepMind, based on Griffin.
  ☆636Updated 2 months ago
• tysam-code / hlb-gpt
  Minimalistic, extremely fast, and hackable researcher's toolbench for GPT models in 307 lines of code. Reaches <3.8 validation loss on wi…
  ☆342Updated 8 months ago
• pytorch-labs / attention-gym
  Helpful tools and examples for working with flex-attention
  ☆720Updated last week
• apple / ml-sigma-reparam
  ☆302Updated 9 months ago
• clu0 / unet.cu
  UNet diffusion model in pure CUDA
  ☆601Updated 9 months ago