lucidrains / nGPT-pytorch
Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAI
☆240Updated this week
Related projects ⓘ
Alternatives and complementary repositories for nGPT-pytorch
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆474Updated 2 weeks ago
- Official Repository of The Mamba in the Llama: Distilling and Accelerating Hybrid Models☆169Updated last week
- Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters☆211Updated this week
- Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793☆323Updated last week
- ☆227Updated 2 months ago
- [ICML 2024 Best Paper] Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (https://arxiv.org/abs/2310.16834)☆394Updated 8 months ago
- Helpful tools and examples for working with flex-attention☆462Updated 2 weeks ago
- Some preliminary explorations of Mamba's context scaling.☆190Updated 9 months ago
- Annotated version of the Mamba paper☆455Updated 8 months ago
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆111Updated 2 months ago
- PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention…☆279Updated 6 months ago
- Official JAX implementation of Learning to (Learn at Test Time): RNNs with Expressive Hidden States☆362Updated 3 months ago
- Implementation of Diffusion Transformer (DiT) in JAX☆252Updated 5 months ago
- Implementation of the proposed Adam-atan2 from Google Deepmind in Pytorch☆94Updated 2 weeks ago
- Implementation of the proposed minGRU in Pytorch☆229Updated 3 weeks ago
- PyTorch implementation of models from the Zamba2 series.☆158Updated this week
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆213Updated 2 months ago
- A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.☆479Updated 2 weeks ago
- Implementation of the Llama architecture with RLHF + Q-learning☆156Updated 10 months ago
- Griffin MQA + Hawk Linear RNN Hybrid☆85Updated 6 months ago
- ☆457Updated 3 months ago
- FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores☆278Updated last month
- Implementation of a multimodal diffusion transformer in Pytorch☆97Updated 4 months ago
- ☆175Updated this week
- Understand and test language model architectures on synthetic tasks.☆161Updated 6 months ago
- Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch☆291Updated 4 months ago
- code for "Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion"☆591Updated 3 weeks ago
- Simplified Masked Diffusion Language Model☆202Updated this week
- Implementation of a single layer of the MMDiT, proposed in Stable Diffusion 3, in Pytorch☆253Updated 2 months ago
- Implementation of Infini-Transformer in Pytorch☆104Updated last month