expz / annotated-hyena
An annotated implementation of the Hyena Hierarchy paper
โ30Updated last year
Related projects: โ
- JAX/Flax implementation of the Hyena Hierarchyโ29Updated last year
- A MAD laboratory to improve AI architecture designs ๐งชโ84Updated 4 months ago
- โ47Updated 3 months ago
- โ42Updated 3 months ago
- โ42Updated this week
- Unofficial but Efficient Implementation of "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" in JAXโ74Updated 7 months ago
- Efficient PScan implementation in PyTorchโ15Updated 8 months ago
- PyTorch implementation for "Long Horizon Temperature Scaling", ICML 2023โ18Updated last year
- Implementation of GateLoop Transformer in Pytorch and Jaxโ86Updated 3 months ago
- Triton Implementation of HyperAttention Algorithmโ46Updated 9 months ago
- My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hโฆโ51Updated last year
- Engineering the state of RNN language models (Mamba, RWKV, etc.)โ31Updated 3 months ago
- Blog postโ16Updated 7 months ago
- โ35Updated 5 months ago
- โ48Updated 4 months ago
- โ28Updated last week
- โ30Updated 8 months ago
- Implementation of Infini-Transformer in Pytorchโ100Updated last month
- Why Do We Need Weight Decay in Modern Deep Learning? [arXiv, Oct 2023]โ41Updated 11 months ago
- โ23Updated this week
- Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amountโฆโ50Updated 10 months ago
- โ22Updated 10 months ago
- Jax implementation of "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"โ12Updated 4 months ago
- โ32Updated 9 months ago
- Source-to-Source Debuggable Derivatives in Pure Pythonโ14Updated 7 months ago
- Code Release for "Broken Neural Scaling Laws" (BNSL) paperโ57Updated 10 months ago
- โ42Updated 7 months ago
- [ICML 24 NGSM workshop] Associative Recurrent Memory Transformer implementation and scripts for training and evaluatingโ26Updated last week
- Sequence Modeling with Structured State Spacesโ60Updated 2 years ago
- Implementation of Gated State Spaces, from the paper "Long Range Language Modeling via Gated State Spaces", in Pytorchโ94Updated last year