bentherien / mu_learned_optimizationLinks
[Oral; Neurips OPT2024 ] μLO: Compute-Efficient Meta-Generalization of Learned Optimizers
☆13Updated 3 months ago
Alternatives and similar repositories for mu_learned_optimization
Users that are interested in mu_learned_optimization are comparing it to the libraries listed below
Sorting:
- Official code for the paper "Attention as a Hypernetwork"☆39Updated last year
- Latest Weight Averaging (NeurIPS HITY 2022)☆30Updated 2 years ago
- Combining SOAP and MUON☆16Updated 4 months ago
- Code for the paper "Function-Space Learning Rates"☆20Updated 3 weeks ago
- Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"☆38Updated last week
- ☆44Updated last year
- ☆34Updated 9 months ago
- [NeurIPS 2023] Sparse Modular Activation for Efficient Sequence Modeling☆37Updated last year
- sigma-MoE layer☆19Updated last year
- Using FlexAttention to compute attention with different masking patterns☆44Updated 9 months ago
- Code for "Accelerating Training with Neuron Interaction and Nowcasting Networks" [to appear at ICLR 2025]☆19Updated last month
- ☆53Updated 8 months ago
- This repo is based on https://github.com/jiaweizzhao/GaLore☆28Updated 9 months ago
- ☆32Updated last year
- Transformer with Mu-Parameterization, implemented in Jax/Flax. Supports FSDP on TPU pods.☆30Updated 2 weeks ago
- ☆32Updated last year
- Reference implementation of "Softmax Attention with Constant Cost per Token" (Heinsen, 2024)☆24Updated last year
- HGRN2: Gated Linear RNNs with State Expansion☆55Updated 10 months ago
- ☆48Updated last year
- ☆51Updated last year
- Blog post☆17Updated last year
- ☆29Updated 2 years ago
- Codes accompanying the paper "LaProp: a Better Way to Combine Momentum with Adaptive Gradient"☆29Updated 4 years ago
- CUDA implementation of autoregressive linear attention, with all the latest research findings☆44Updated 2 years ago
- ☆27Updated last year
- Here we will test various linear attention designs.☆59Updated last year
- Efficient Scaling laws and collaborative pretraining.☆16Updated 4 months ago
- ☆32Updated last year
- Experimental scripts for researching data adaptive learning rate scheduling.☆23Updated last year
- Implementation of Spectral State Space Models☆16Updated last year