davisyoshida / abnormal-floats
Code for the note "NF4 Isn't Information Theoretically Optimal (and that's Good)
☆18Updated last year
Related projects ⓘ
Alternatives and complementary repositories for abnormal-floats
- [NeurIPS 2023] Sparse Modular Activation for Efficient Sequence Modeling☆35Updated 11 months ago
- Engineering the state of RNN language models (Mamba, RWKV, etc.)☆32Updated 5 months ago
- ☆50Updated last week
- ☆31Updated 10 months ago
- sigma-MoE layer☆18Updated 10 months ago
- ☆44Updated 4 months ago
- Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount…☆49Updated last year
- Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"☆36Updated 11 months ago
- ☆45Updated 9 months ago
- ☆31Updated 2 months ago
- ☆76Updated 5 months ago
- See https://github.com/cuda-mode/triton-index/ instead!☆11Updated 6 months ago
- Simple and efficient pytorch-native transformer training and inference (batched)☆61Updated 7 months ago
- ☆18Updated 5 months ago
- Using FlexAttention to compute attention with different masking patterns☆40Updated last month
- ☆25Updated 11 months ago
- GoldFinch and other hybrid transformer components☆39Updated 3 months ago
- ☆20Updated last year
- ☆44Updated last year
- Implementation of Hyena Hierarchy in JAX☆10Updated last year
- ☆21Updated last month
- Minimum Description Length probing for neural network representations☆16Updated last week
- Awesome Triton Resources☆18Updated 3 weeks ago
- My explorations into editing the knowledge and memories of an attention network☆34Updated last year
- ☆76Updated 6 months ago
- Here we will test various linear attention designs.☆56Updated 6 months ago
- [ICML 24 NGSM workshop] Associative Recurrent Memory Transformer implementation and scripts for training and evaluating☆31Updated this week
- Token Omission Via Attention☆119Updated 3 weeks ago
- Parallel Associative Scan for Language Models☆18Updated 10 months ago