EleutherAI / rnngineering
Engineering the state of RNN language models (Mamba, RWKV, etc.)
☆32Updated 9 months ago
Alternatives and similar repositories for rnngineering:
Users that are interested in rnngineering are comparing it to the libraries listed below
- Minimum Description Length probing for neural network representations☆19Updated last month
- ☆19Updated 5 months ago
- ☆33Updated last year
- Source-to-Source Debuggable Derivatives in Pure Python☆15Updated last year
- Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing☆48Updated 3 years ago
- Triton Implementation of HyperAttention Algorithm☆47Updated last year
- Experiments on the impact of depth in transformers and SSMs.☆23Updated 4 months ago
- Implementation of Gradient Agreement Filtering, from Chaubard et al. of Stanford, but for single machine microbatches, in Pytorch☆23Updated last month
- LayerNorm(SmallInit(Embedding)) in a Transformer to improve convergence☆60Updated 3 years ago
- Blog post☆17Updated last year
- ☆33Updated 6 months ago
- Here we will test various linear attention designs.☆60Updated 10 months ago
- RWKV model implementation☆37Updated last year
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆26Updated 11 months ago
- [NeurIPS 2023] Sparse Modular Activation for Efficient Sequence Modeling☆36Updated last year
- My explorations into editing the knowledge and memories of an attention network☆34Updated 2 years ago
- ☆21Updated 3 months ago
- Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.☆17Updated this week
- ☆18Updated 10 months ago
- Aioli: A unified optimization framework for language model data mixing☆22Updated 2 months ago
- Implementation of a holodeck, written in Pytorch☆17Updated last year
- GoldFinch and other hybrid transformer components☆45Updated 7 months ago
- Official Repository for Efficient Linear-Time Attention Transformers.☆18Updated 9 months ago
- Latent Diffusion Language Models☆68Updated last year
- ☆18Updated 9 months ago
- Transformer with Mu-Parameterization, implemented in Jax/Flax. Supports FSDP on TPU pods.☆30Updated 3 months ago
- See https://github.com/cuda-mode/triton-index/ instead!☆11Updated 10 months ago
- Efficient Scaling laws and collaborative pretraining.☆15Updated last month
- ☆30Updated 5 months ago