huyphan168 / PEER
Mixture of A Million Experts
☆42Updated 8 months ago
Alternatives and similar repositories for PEER:
Users that are interested in PEER are comparing it to the libraries listed below
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆122Updated 7 months ago
- ☆74Updated 7 months ago
- Griffin MQA + Hawk Linear RNN Hybrid☆85Updated 11 months ago
- Token Omission Via Attention☆124Updated 5 months ago
- This repo is based on https://github.com/jiaweizzhao/GaLore☆26Updated 6 months ago
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆224Updated last month
- Language models scale reliably with over-training and on downstream tasks☆96Updated last year
- Stick-breaking attention☆49Updated 2 weeks ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆125Updated 4 months ago
- Understand and test language model architectures on synthetic tasks.☆185Updated 3 weeks ago
- ☆87Updated 6 months ago
- ☆81Updated last year
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆97Updated 6 months ago
- ☆51Updated 10 months ago
- ☆65Updated last month
- Official implementation of Phi-Mamba. A MOHAWK-distilled model (Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Mode…☆102Updated 6 months ago
- ☆67Updated 8 months ago
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆123Updated 11 months ago
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆103Updated 4 months ago
- Implementation of Infini-Transformer in Pytorch☆110Updated 2 months ago
- ☆76Updated 8 months ago
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)☆150Updated 3 months ago
- Some preliminary explorations of Mamba's context scaling.☆212Updated last year
- Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch☆161Updated 3 months ago
- Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"☆36Updated last year
- [ICLR 2025] Official PyTorch implementation of "Forgetting Transformer: Softmax Attention with a Forget Gate"☆80Updated last week
- Tiny re-implementation of MDM in style of LLaDA and nano-gpt speedrun☆44Updated 3 weeks ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"☆59Updated 5 months ago
- ☆125Updated last year
- The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.☆59Updated 2 months ago