shawntan / scattermoe
Triton-based implementation of Sparse Mixture of Experts.
☆184Updated 3 weeks ago
Related projects ⓘ
Alternatives and complementary repositories for scattermoe
- Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers☆196Updated 2 months ago
- ☆183Updated 6 months ago
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆190Updated 2 weeks ago
- This repository contains the experimental PyTorch native float8 training UX☆211Updated 3 months ago
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆146Updated 3 months ago
- ☆88Updated 2 months ago
- Explorations into some recent techniques surrounding speculative decoding☆209Updated last year
- ☆132Updated last year
- 🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.☆163Updated this week
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆212Updated 2 months ago
- ☆95Updated last month
- REST: Retrieval-Based Speculative Decoding, NAACL 2024☆172Updated last month
- Fast Matrix Multiplications for Lookup Table-Quantized LLMs☆183Updated last month
- Cataloging released Triton kernels.☆132Updated 2 months ago
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆195Updated last week
- ring-attention experiments☆95Updated 3 weeks ago
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆277Updated 4 months ago
- Applied AI experiments and examples for PyTorch☆159Updated last week
- Boosting 4-bit inference kernels with 2:4 Sparsity☆51Updated 2 months ago
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆240Updated 3 weeks ago
- Triton implementation of FlashAttention2 that adds Custom Masks.☆72Updated 2 months ago
- ☆140Updated this week
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆70Updated this week
- Simple and fast low-bit matmul kernels in CUDA / Triton☆133Updated this week
- [NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization☆302Updated 2 months ago
- extensible collectives library in triton☆61Updated last month
- Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of…☆86Updated 3 months ago
- An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).☆196Updated last week
- ☆34Updated 8 months ago
- Code for Palu: Compressing KV-Cache with Low-Rank Projection☆54Updated this week