CLAIRE-Labo / StructuredFFN
The official code of "Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers"
☆14Updated 3 months ago
Related projects ⓘ
Alternatives and complementary repositories for StructuredFFN
- Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"☆36Updated 11 months ago
- Code Release for "Broken Neural Scaling Laws" (BNSL) paper☆57Updated last year
- ☆50Updated 5 months ago
- ☆27Updated 7 months ago
- ☆23Updated 8 months ago
- Minimal but scalable implementation of large language models in JAX☆25Updated last week
- ☆29Updated last month
- Efficient PScan implementation in PyTorch☆15Updated 10 months ago
- Code for minimum-entropy coupling.☆29Updated 4 months ago
- ☆46Updated last month
- ☆53Updated 9 months ago
- ☆31Updated 10 months ago
- Engineering the state of RNN language models (Mamba, RWKV, etc.)☆32Updated 5 months ago
- Blog post☆16Updated 8 months ago
- ☆45Updated 9 months ago
- ☆18Updated last month
- Parallel Associative Scan for Language Models☆18Updated 10 months ago
- Using FlexAttention to compute attention with different masking patterns☆40Updated last month
- ☆44Updated 4 months ago
- A State-Space Model with Rational Transfer Function Representation.☆70Updated 5 months ago
- RWKV model implementation☆38Updated last year
- ☆9Updated 10 months ago
- ☆16Updated 2 months ago
- Unofficial but Efficient Implementation of "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" in JAX☆78Updated 9 months ago
- ☆24Updated 8 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆24Updated 6 months ago
- ☆36Updated 10 months ago
- ☆50Updated 2 weeks ago
- Code to train and evaluate Neural Attention Memory Models to obtain universally-applicable memory systems for transformers.☆11Updated 3 weeks ago