davisyoshida / abnormal-floats
Code for the note "NF4 Isn't Information Theoretically Optimal (and that's Good)
☆18Updated last year
Related projects ⓘ
Alternatives and complementary repositories for abnormal-floats
- Engineering the state of RNN language models (Mamba, RWKV, etc.)☆32Updated 5 months ago
- sigma-MoE layer☆18Updated 10 months ago
- ☆53Updated 3 weeks ago
- Efficient Scaling laws and collaborative pretraining.☆13Updated this week
- ☆53Updated 10 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆24Updated 7 months ago
- ☆25Updated 11 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"☆56Updated last month
- See https://github.com/cuda-mode/triton-index/ instead!☆11Updated 6 months ago
- Blog post☆16Updated 9 months ago
- ☆31Updated 10 months ago
- Efficient PScan implementation in PyTorch☆15Updated 10 months ago
- Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount…☆49Updated last year
- ☆50Updated 6 months ago
- ☆28Updated 7 months ago
- Minimum Description Length probing for neural network representations☆16Updated last week
- Transformer with Mu-Parameterization, implemented in Jax/Flax. Supports FSDP on TPU pods.☆29Updated 2 weeks ago
- ☆46Updated last month
- Utilities for Training Very Large Models☆56Updated last month
- Here we will test various linear attention designs.☆56Updated 6 months ago
- ☆77Updated 5 months ago
- A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.☆54Updated 3 months ago
- Stick-breaking attention☆34Updated last week
- [NeurIPS 2023] Sparse Modular Activation for Efficient Sequence Modeling☆35Updated 11 months ago
- ☆45Updated 9 months ago
- Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"☆36Updated last year
- Using FlexAttention to compute attention with different masking patterns☆40Updated 2 months ago
- ☆29Updated 2 months ago
- Fast, Modern, Memory Efficient, and Low Precision PyTorch Optimizers☆58Updated 4 months ago
- Awesome Triton Resources☆18Updated last month