EleutherAI / rnngineering
Engineering the state of RNN language models (Mamba, RWKV, etc.)
☆32Updated 9 months ago
Alternatives and similar repositories for rnngineering:
Users that are interested in rnngineering are comparing it to the libraries listed below
- Minimum Description Length probing for neural network representations☆19Updated last month
- ☆33Updated last year
- ☆18Updated 9 months ago
- Source-to-Source Debuggable Derivatives in Pure Python☆15Updated last year
- Blog post☆17Updated last year
- Triton Implementation of HyperAttention Algorithm☆47Updated last year
- ☆19Updated 5 months ago
- Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing☆48Updated 3 years ago
- My explorations into editing the knowledge and memories of an attention network☆34Updated 2 years ago
- Fine-Tuning Pre-trained Transformers into Decaying Fast Weights☆19Updated 2 years ago
- [NeurIPS 2023] Sparse Modular Activation for Efficient Sequence Modeling☆36Updated last year
- JAX/Flax implementation of the Hyena Hierarchy☆34Updated last year
- Awesome Triton Resources☆20Updated 3 months ago
- Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.☆17Updated this week
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆25Updated 10 months ago
- RWKV model implementation☆37Updated last year
- Transformer with Mu-Parameterization, implemented in Jax/Flax. Supports FSDP on TPU pods.☆30Updated 2 months ago
- Parallel Associative Scan for Language Models☆18Updated last year
- See https://github.com/cuda-mode/triton-index/ instead!☆11Updated 10 months ago
- GoldFinch and other hybrid transformer components☆45Updated 7 months ago
- PyTorch implementation for "Long Horizon Temperature Scaling", ICML 2023☆20Updated last year
- Implementation of Spectral State Space Models☆16Updated last year
- ☆30Updated 5 months ago
- LayerNorm(SmallInit(Embedding)) in a Transformer to improve convergence☆60Updated 3 years ago
- DiCE: The Infinitely Differentiable Monte-Carlo Estimator☆31Updated last year
- ☆33Updated 6 months ago
- ☆21Updated 3 months ago