lucidrains / PEER-pytorch
Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind
☆112Updated 2 months ago
Related projects ⓘ
Alternatives and complementary repositories for PEER-pytorch
- Some preliminary explorations of Mamba's context scaling.☆191Updated 9 months ago
- Mixture of A Million Experts☆32Updated 3 months ago
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆92Updated last month
- Griffin MQA + Hawk Linear RNN Hybrid☆85Updated 6 months ago
- Normalized Transformer (nGPT)☆66Updated this week
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆214Updated 3 months ago
- Token Omission Via Attention☆120Updated last month
- ☆175Updated this week
- Understand and test language model architectures on synthetic tasks.☆162Updated 6 months ago
- Implementation of Infini-Transformer in Pytorch☆104Updated last month
- A byte-level decoder architecture that matches the performance of tokenized Transformers.☆59Updated 6 months ago
- Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts☆108Updated last month
- Here we will test various linear attention designs.☆56Updated 6 months ago
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆113Updated 7 months ago
- [NeurIPS 2024] Official Repository of The Mamba in the Llama: Distilling and Accelerating Hybrid Models☆174Updated this week
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (Official Code)☆135Updated last month
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆104Updated last month
- ☆62Updated 3 months ago
- Explorations into the recently proposed Taylor Series Linear Attention☆90Updated 3 months ago
- Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…☆78Updated 2 months ago
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆84Updated last week
- A MAD laboratory to improve AI architecture designs 🧪☆95Updated 6 months ago
- ☆73Updated 4 months ago
- ☆132Updated last year
- Simple implementation of muP, based on Spectral Condition for Feature Learning. The implementation is SGD only, dont use it for Adam☆69Updated 3 months ago
- Implementation of a multimodal diffusion transformer in Pytorch☆97Updated 4 months ago
- Exploration into the proposed "Self Reasoning Tokens" by Felipe Bonetto☆53Updated 6 months ago
- ☆122Updated 9 months ago
- ☆53Updated 10 months ago