ambisinister / mla-experimentsLinks
Experiments on Multi-Head Latent Attention
☆95Updated last year
Alternatives and similar repositories for mla-experiments
Users that are interested in mla-experiments are comparing it to the libraries listed below
Sorting:
- Flash-Muon: An Efficient Implementation of Muon Optimizer☆186Updated 3 months ago
- The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.☆87Updated 2 months ago
- The evaluation framework for training-free sparse attention in LLMs☆96Updated 3 months ago
- ☆126Updated 3 months ago
- Fast and memory-efficient exact attention☆69Updated 6 months ago
- ☆83Updated last year
- Get down and dirty with FlashAttention2.0 in pytorch, plug in and play no complex CUDA kernels☆109Updated 2 years ago
- Awesome Triton Resources☆33Updated 4 months ago
- Code for studying the super weight in LLM☆117Updated 9 months ago
- Transformers components but in Triton☆34Updated 4 months ago
- ☆118Updated last year
- FlashRNN - Fast RNN Kernels with I/O Awareness☆97Updated 3 months ago
- Implementation of Infini-Transformer in Pytorch☆111Updated 8 months ago
- ☆85Updated 6 months ago
- Muon fsdp 2☆43Updated last month
- CUDA and Triton implementations of Flash Attention with SoftmaxN.☆73Updated last year
- Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆171Updated last year
- ☆149Updated 2 years ago
- Low-bit optimizers for PyTorch☆131Updated last year
- Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆106Updated last week
- From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…☆48Updated 5 months ago
- ☆56Updated last year
- An unofficial implementation of "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆36Updated last year
- ring-attention experiments☆152Updated 11 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆60Updated 11 months ago
- SQUEEZED ATTENTION: Accelerating Long Prompt LLM Inference☆53Updated 10 months ago
- Simple and efficient pytorch-native transformer training and inference (batched)☆78Updated last year
- Triton-based implementation of Sparse Mixture of Experts.☆240Updated 3 weeks ago
- 🔥 A minimal training framework for scaling FLA models☆247Updated last week
- ☆142Updated 7 months ago