User-friendly implementation of the Mixture-of-Sparse-Attention (MoSA). MoSA selects distinct tokens for each head with expert choice routing providing a content-based sparse attention mechanism.
☆30May 3, 2025Updated last year
Alternatives and similar repositories for MoSA
Users that are interested in MoSA are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- The official implementation of HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization☆19Mar 7, 2025Updated last year
- [TMLR] "Can You Win Everything with Lottery Ticket?" by Tianlong Chen, Zhenyu Zhang, Jun Wu, Randy Huang, Sijia Liu, Shiyu Chang, Zhangy…☆10Sep 8, 2022Updated 4 years ago
- sigma-MoE layer☆21Jan 5, 2024Updated 2 years ago
- Official implementation of Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models (ICLR 2024 Spotlight)☆16Jul 19, 2026Updated 3 months ago
- ☆12Apr 29, 2024Updated 2 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- ☆14Oct 3, 2024Updated 2 years ago
- [ICML 2025] LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models☆16Nov 4, 2025Updated 11 months ago
- Community Implementation of the paper: "Multi-Head Mixture-of-Experts" In PyTorch☆31Updated this week
- MobileLLM-R1☆95Apr 30, 2026Updated 6 months ago
- ☆16May 24, 2026Updated 5 months ago
- ☆139May 29, 2025Updated last year
- ☆18Dec 2, 2024Updated last year
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS'25 Spotlight)☆27Sep 25, 2026Updated last month
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆102Sep 30, 2024Updated 2 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Revealing and unlocking the context boundary of reward models☆21May 10, 2026Updated 5 months ago
- ☆10Jul 27, 2026Updated 3 months ago
- Activation-Steered Compression☆18Jan 30, 2026Updated 9 months ago
- ☆14Jul 13, 2025Updated last year
- ☆13Aug 13, 2024Updated 2 years ago
- Extensive Self-Contrast Enables Feedback-Free Language Model Alignment☆20Apr 2, 2024Updated 2 years ago
- ☆20Jan 10, 2025Updated last year
- Don't just regulate gradients like in Muon, regulate the weights too☆32Jul 30, 2025Updated last year
- ☆12Oct 21, 2026Updated last week
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing (Technical Report)☆39Jul 10, 2026Updated 3 months ago
- KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation, NAACL 2024☆16Jul 29, 2024Updated 2 years ago
- ☆28Jul 2, 2026Updated 3 months ago
- Minimal implementation of the Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models paper (ArXiv 20232401.01335)☆29Mar 1, 2024Updated 2 years ago
- Official Pytorch implementation of Chromatic Graph Transformers☆10Jun 14, 2023Updated 3 years ago
- ☆10Jun 11, 2024Updated 2 years ago
- [NeurIPS 2024] | An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding☆23Oct 10, 2024Updated 2 years ago
- Clustered Compositional Embeddings☆13Oct 25, 2023Updated 3 years ago
- RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment☆18Dec 19, 2024Updated last year
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- [EMNLP 2024 Findings] Unlocking Continual Learning Abilities in Language Models☆26Oct 8, 2024Updated 2 years ago
- Fast, Numerically Stable, and Auto-Differentiable Spectral Clipping via Newton-Schulz Iteration☆12Jun 21, 2025Updated last year
- [ICML 2024] SIRFShampoo: Structured inverse- and root-free Shampoo in PyTorch (https://arxiv.org/abs/2402.03496)☆15Nov 4, 2024Updated last year
- ☆13Apr 16, 2024Updated 2 years ago
- ☆11Apr 13, 2024Updated 2 years ago
- Code to reproduce key results for "Interpreting Attention Layer Outputs with Sparse Autoencoders"☆11Jul 27, 2024Updated 2 years ago
- Least Squares Regression for subspace clustering☆11May 27, 2018Updated 8 years ago