LIONS-EPFL / scion
☆14Updated last month
Alternatives and similar repositories for scion:
Users that are interested in scion are comparing it to the libraries listed below
- Combining SOAP and MUON☆15Updated 2 months ago
- ☆19Updated 3 weeks ago
- Code for the paper "Function-Space Learning Rates"☆19Updated last week
- ☆31Updated last year
- Tiny re-implementation of MDM in style of LLaDA and nano-gpt speedrun☆48Updated last month
- ☆31Updated last year
- ☆51Updated 11 months ago
- ☆33Updated 7 months ago
- Here we will test various linear attention designs.☆60Updated last year
- Simple implementation of muP, based on Spectral Condition for Feature Learning. The implementation is SGD only, dont use it for Adam☆75Updated 8 months ago
- ☆52Updated 6 months ago
- Efficient PScan implementation in PyTorch☆16Updated last year
- ☆30Updated 5 months ago
- This repo is based on https://github.com/jiaweizzhao/GaLore☆26Updated 7 months ago
- The official code of "Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers"☆19Updated 9 months ago
- ☆60Updated 5 months ago
- research impl of Native Sparse Attention (2502.11089)☆53Updated 2 months ago
- Blog post☆17Updated last year
- Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.☆17Updated last month
- Triton Implementation of HyperAttention Algorithm☆47Updated last year
- ☆18Updated 9 months ago
- GoldFinch and other hybrid transformer components☆45Updated 9 months ago
- Transformer with Mu-Parameterization, implemented in Jax/Flax. Supports FSDP on TPU pods.☆30Updated this week
- Reference implementation of "Softmax Attention with Constant Cost per Token" (Heinsen, 2024)☆24Updated 10 months ago
- ☆32Updated last year
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆26Updated last year
- Tiled Flash Linear Attention library for fast and efficient mLSTM Kernels.☆54Updated 2 weeks ago
- Unofficial Implementation of Selective Attention Transformer☆16Updated 5 months ago
- Codes accompanying the paper "LaProp: a Better Way to Combine Momentum with Adaptive Gradient"☆28Updated 4 years ago
- Implementation of GateLoop Transformer in Pytorch and Jax☆87Updated 10 months ago