User-friendly implementation of the Mixture-of-Sparse-Attention (MoSA). MoSA selects distinct tokens for each head with expert choice routing providing a content-based sparse attention mechanism.
☆30May 3, 2025Updated last year
Alternatives and similar repositories for MoSA
Users that are interested in MoSA are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- The official implementation of HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization☆19Mar 7, 2025Updated last year
- [TMLR] "Can You Win Everything with Lottery Ticket?" by Tianlong Chen, Zhenyu Zhang, Jun Wu, Randy Huang, Sijia Liu, Shiyu Chang, Zhangy…☆10Sep 8, 2022Updated 4 years ago
- sigma-MoE layer☆21Jan 5, 2024Updated 2 years ago
- Official implementation of Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models (ICLR 2024 Spotlight)☆16Jul 19, 2026Updated 4 months ago
- ☆12Apr 29, 2024Updated 2 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- ☆14Oct 3, 2024Updated 2 years ago
- [ICML 2025] LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models☆16Nov 4, 2025Updated last year
- ☆32Apr 22, 2026Updated 6 months ago
- ☆139May 29, 2025Updated last year
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS'25 Spotlight)☆27Sep 25, 2026Updated last month
- ☆16Jun 4, 2025Updated last year
- ☆14Mar 20, 2025Updated last year
- Revealing and unlocking the context boundary of reward models☆21May 10, 2026Updated 6 months ago
- Official implementation of "MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning"☆11Aug 23, 2025Updated last year
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Activation-Steered Compression☆17Jan 30, 2026Updated 9 months ago
- ☆14Jul 13, 2025Updated last year
- Extensive Self-Contrast Enables Feedback-Free Language Model Alignment☆20Apr 2, 2024Updated 2 years ago
- Dynamical low rank training for neural networks☆11Aug 24, 2023Updated 3 years ago
- A training-free method for extending the maximum context length of language models☆14Oct 6, 2025Updated last year
- ☆20Jan 10, 2025Updated last year
- ☆20Jan 24, 2025Updated last year
- ☆28Jul 2, 2026Updated 4 months ago
- codes and plots for "Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs"☆11Dec 30, 2024Updated last year
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Minimal implementation of the Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models paper (ArXiv 20232401.01335)☆29Mar 1, 2024Updated 2 years ago
- ☆16May 24, 2026Updated 5 months ago
- RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment☆18Dec 19, 2024Updated last year
- [NeurIPS 2024] | An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding☆23Oct 10, 2024Updated 2 years ago
- Official Pytorch implementation of Chromatic Graph Transformers☆10Jun 14, 2023Updated 3 years ago
- [EMNLP 2024 Findings] Unlocking Continual Learning Abilities in Language Models☆25Oct 8, 2024Updated 2 years ago
- Clustered Compositional Embeddings☆13Oct 25, 2023Updated 3 years ago
- ☆14Aug 13, 2024Updated 2 years ago
- [ICLR'26] Beyond Linear Probes: Dynamic Safety Monitoring for Language Models☆13Oct 1, 2025Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Fast, Numerically Stable, and Auto-Differentiable Spectral Clipping via Newton-Schulz Iteration☆12Jun 21, 2025Updated last year
- ☆27May 12, 2026Updated 6 months ago
- Least Squares Regression for subspace clustering☆11May 27, 2018Updated 8 years ago
- ☆17Jun 11, 2025Updated last year
- Elevating Chess Strategy with Fine-Tuned Large Language Model☆20Dec 8, 2023Updated 2 years ago
- Codes for our paper "AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems"☆14Dec 13, 2024Updated last year
- ☆29Apr 7, 2024Updated 2 years ago
Are these results useful?