insuhan / hyper-attn
☆81Updated last year
Alternatives and similar repositories for hyper-attn:
Users that are interested in hyper-attn are comparing it to the libraries listed below
- ☆139Updated last year
- The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.☆58Updated last month
- Understand and test language model architectures on synthetic tasks.☆184Updated last week
- A MAD laboratory to improve AI architecture designs 🧪☆107Updated 3 months ago
- Fast and memory-efficient exact attention☆66Updated 2 weeks ago
- ☆51Updated 9 months ago
- ☆49Updated 4 months ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆123Updated 3 months ago
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆71Updated 4 months ago
- ☆125Updated last year
- Code for studying the super weight in LLM☆91Updated 3 months ago
- ☆115Updated last month
- ☆94Updated 9 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"☆59Updated 5 months ago
- Triton-based implementation of Sparse Mixture of Experts.☆207Updated 3 months ago
- Stick-breaking attention☆48Updated this week
- 🔥 A minimal training framework for scaling FLA models☆79Updated this week
- Language models scale reliably with over-training and on downstream tasks☆96Updated 11 months ago
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆223Updated last month
- [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"☆74Updated 9 months ago
- Sparse Backpropagation for Mixture-of-Expert Training☆28Updated 8 months ago
- Experiment of using Tangent to autodiff triton☆76Updated last year
- [NeurIPS'23] Speculative Decoding with Big Little Decoder☆89Updated last year
- Activation-aware Singular Value Decomposition for Compressing Large Language Models☆58Updated 4 months ago
- A fast implementation of T5/UL2 in PyTorch using Flash Attention☆85Updated last week
- ☆42Updated last year
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆100Updated 4 months ago
- nanoGPT-like codebase for LLM training☆90Updated this week
- Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of…☆122Updated 7 months ago