lucidrains / nGPT-pytorchLinks
Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAI
☆291Updated 4 months ago
Alternatives and similar repositories for nGPT-pytorch
Users that are interested in nGPT-pytorch are comparing it to the libraries listed below
Sorting:
- Normalized Transformer (nGPT)☆192Updated 11 months ago
- Official implementation of the paper: "ZClip: Adaptive Spike Mitigation for LLM Pre-Training".☆135Updated last week
- Some preliminary explorations of Mamba's context scaling.☆216Updated last year
- [ICLR 2025] Official PyTorch Implementation of Gated Delta Networks: Improving Mamba2 with Delta Rule☆331Updated last month
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆129Updated last year
- ☆302Updated 6 months ago
- Annotated version of the Mamba paper☆489Updated last year
- ☆251Updated 4 months ago
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆248Updated 8 months ago
- Code accompanying the paper "Generalized Interpolating Discrete Diffusion"☆103Updated 4 months ago
- Implementation of the Llama architecture with RLHF + Q-learning☆167Updated 8 months ago
- DeMo: Decoupled Momentum Optimization☆194Updated 10 months ago
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆241Updated 4 months ago
- The AdEMAMix Optimizer: Better, Faster, Older.☆186Updated last year
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆542Updated 5 months ago
- Flash Attention Triton kernel with support for second-order derivatives☆103Updated this week
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆166Updated 3 months ago
- ☆200Updated last month
- Getting crystal-like representations with harmonic loss☆193Updated 6 months ago
- [ICLR2025 Spotlight🔥] Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters☆574Updated 8 months ago
- Understand and test language model architectures on synthetic tasks.☆233Updated last month
- CIFAR-10 speedruns: 94% in 2.6 seconds and 96% in 27 seconds☆313Updated 3 months ago
- Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793☆439Updated 5 months ago
- Attempt to make multiple residual streams from Bytedance's Hyper-Connections paper accessible to the public☆91Updated 4 months ago
- A MAD laboratory to improve AI architecture designs 🧪☆131Updated 10 months ago
- Implementation of the proposed Adam-atan2 from Google Deepmind in Pytorch☆129Updated last week
- ☆201Updated 10 months ago
- Accelerated First Order Parallel Associative Scan☆189Updated last year
- Flash-Muon: An Efficient Implementation of Muon Optimizer☆195Updated 4 months ago
- Implementation of Diffusion Transformer (DiT) in JAX☆295Updated last year