lucidrains / nGPT-pytorch
Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAI
☆256Updated 2 weeks ago
Related projects ⓘ
Alternatives and complementary repositories for nGPT-pytorch
- Annotated version of the Mamba paper☆457Updated 8 months ago
- Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters☆335Updated last week
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆476Updated 3 weeks ago
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆112Updated 2 months ago
- Some preliminary explorations of Mamba's context scaling.☆191Updated 9 months ago
- Griffin MQA + Hawk Linear RNN Hybrid☆85Updated 6 months ago
- Simplified Masked Diffusion Language Model☆207Updated last week
- Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793☆328Updated 3 weeks ago
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆214Updated 3 months ago
- A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.☆483Updated 3 weeks ago
- [NeurIPS 2024] Official Repository of The Mamba in the Llama: Distilling and Accelerating Hybrid Models☆174Updated this week
- ☆228Updated 2 months ago
- Helpful tools and examples for working with flex-attention☆469Updated 3 weeks ago
- Implementation of Diffusion Transformer (DiT) in JAX☆252Updated 5 months ago
- Understand and test language model architectures on synthetic tasks.☆162Updated 6 months ago
- ☆175Updated this week
- Implementation of the Llama architecture with RLHF + Q-learning☆157Updated 10 months ago
- PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention…☆280Updated 6 months ago
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆177Updated last month
- ☆197Updated 4 months ago
- Repo for "Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture"☆537Updated 6 months ago
- Normalized Transformer (nGPT)☆66Updated this week
- Token Omission Via Attention☆120Updated last month
- Implementation of a multimodal diffusion transformer in Pytorch☆97Updated 4 months ago
- PyTorch implementation of models from the Zamba2 series.☆158Updated this week
- PyTorch Implementation of Jamba: "Jamba: A Hybrid Transformer-Mamba Language Model"☆137Updated last week
- Accelerated First Order Parallel Associative Scan☆163Updated 3 months ago
- Implementation of Infini-Transformer in Pytorch☆104Updated last month
- For optimization algorithm research and development.☆449Updated this week
- A MAD laboratory to improve AI architecture designs 🧪☆95Updated 6 months ago