m-a-n-i-f-e-s-t / retentionLinks
Language modeling with linear-cost context
☆115Updated 4 months ago
Alternatives and similar repositories for retention
Users that are interested in retention are comparing it to the libraries listed below
Sorting:
- Supporting code for the blog post on modular manifolds.☆113Updated 4 months ago
- DeMo: Decoupled Momentum Optimization☆198Updated last year
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆133Updated 2 months ago
- ☆82Updated last year
- PyTorch implementation of models from the Zamba2 series.☆186Updated last year
- Tiled Flash Linear Attention library for fast and efficient mLSTM Kernels.☆82Updated 2 months ago
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆132Updated last year
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆102Updated last year
- My fork os allen AI's OLMo for educational purposes.☆29Updated last year
- NanoGPT-speedrunning for the poor T4 enjoyers☆73Updated 9 months ago
- RWKV-7: Surpassing GPT☆104Updated last year
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆131Updated last year
- ☆47Updated 2 years ago
- The Automated LLM Speedrunning Benchmark measures how well LLM agents can reproduce previous innovations and discover new ones in languag…☆127Updated 3 months ago
- A collection of lightweight interpretability scripts to understand how LLMs think☆89Updated last week
- Utilities for Training Very Large Models☆58Updated last year
- ☆92Updated last year
- Focused on fast experimentation and simplicity☆80Updated last year
- Landing repository for the paper "Softpick: No Attention Sink, No Massive Activations with Rectified Softmax"☆86Updated 4 months ago
- Implementation of the Llama architecture with RLHF + Q-learning☆170Updated 11 months ago
- Explorations into the proposal from the paper "Grokfast, Accelerated Grokking by Amplifying Slow Gradients"☆103Updated last year
- Using FlexAttention to compute attention with different masking patterns☆47Updated last year
- ☆91Updated last year
- ☆71Updated last year
- Triton Implementation of HyperAttention Algorithm☆48Updated 2 years ago
- MoE training for Me and You and maybe other people☆331Updated 3 weeks ago
- Implementation of the general framework for AMIE, from the paper "Towards Conversational Diagnostic AI", out of Google Deepmind☆72Updated last year
- supporting pytorch FSDP for optimizers☆84Updated last year
- research impl of Native Sparse Attention (2502.11089)☆63Updated 11 months ago
- Mixture of A Million Experts☆53Updated last year