alexzhang13 / flashattention2-custom-maskLinks
Triton implementation of FlashAttention2 that adds Custom Masks.
β128Updated 11 months ago
Alternatives and similar repositories for flashattention2-custom-mask
Users that are interested in flashattention2-custom-mask are comparing it to the libraries listed below
Sorting:
- Efficient triton implementation of Native Sparse Attention.β186Updated 2 months ago
- π₯ A minimal training framework for scaling FLA modelsβ220Updated last month
- [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Trainingβ221Updated last month
- β123Updated 2 months ago
- β137Updated 5 months ago
- Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Trainingβ212Updated 11 months ago
- Flash-Muon: An Efficient Implementation of Muon Optimizerβ149Updated last month
- Triton-based implementation of Sparse Mixture of Experts.β230Updated 8 months ago
- Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"β119Updated last year
- Odysseus: Playground of LLM Sequence Parallelismβ72Updated last year
- Efficient 2:4 sparse training algorithms and implementationsβ56Updated 7 months ago
- β147Updated 2 years ago
- [ICML 2025] XAttention: Block Sparse Attention with Antidiagonal Scoringβ213Updated last month
- β79Updated 5 months ago
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferenceβ311Updated 3 weeks ago
- A sparse attention kernel supporting mix sparse patternsβ262Updated 5 months ago
- β232Updated 2 months ago
- PyTorch bindings for CUTLASS grouped GEMM.β106Updated 2 months ago
- 16-fold memory access reduction with nearly no lossβ103Updated 4 months ago
- π Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flashβ¦β258Updated last week
- [ICML 2025] SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsityβ48Updated last month
- [ICLR2025] Codebase for "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing", built on Megatron-LM.β85Updated 7 months ago
- VeOmni: Scaling any Modality Model Training to any Accelerators with PyTorch native Training Frameworkβ399Updated this week
- Low-bit optimizers for PyTorchβ130Updated last year
- β113Updated 2 months ago
- Accelerate LLM preference tuning via prefix sharing with a single line of codeβ42Updated last month
- Official implementation of "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding"β320Updated this week
- β268Updated 3 weeks ago
- Fast and memory-efficient exact attentionβ69Updated 5 months ago
- The evaluation framework for training-free sparse attention in LLMsβ86Updated last month