Alternatives and similar repositories for moe_attention

Users that are interested in moe_attention are comparing it to the libraries listed below

• RobertCsordas / moe

  View on GitHub
  Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"
  ☆38Jun 11, 2025Updated 8 months ago
• RobertCsordas / moe_layer

  View on GitHub
  sigma-MoE layer
  ☆21Jan 5, 2024Updated 2 years ago
• jenni-ai / T2FW

  View on GitHub
  Fine-Tuning Pre-trained Transformers into Decaying Fast Weights
  ☆19Oct 9, 2022Updated 3 years ago
• RobertCsordas / switchhead

  View on GitHub
  ☆17Jun 11, 2025Updated 8 months ago
• bdusell / stack-attention

  View on GitHub
  Code for the paper "Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns"
  ☆18Mar 15, 2024Updated last year
• lyutyuh / structured-span-selector

  View on GitHub
  A Structured Span Selector (NAACL 2022). A structured span selector with a WCFG for span selection tasks (coreference resolution, semanti…
  ☆21Jul 11, 2022Updated 3 years ago
• IBM / ModuleFormer

  View on GitHub
  ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward exp…
  ☆226Sep 18, 2025Updated 4 months ago
• ReiherGroup / CoRe_optimizer

  View on GitHub
  Continual Resilient (CoRe) Optimizer for PyTorch
  ☆11Jun 10, 2024Updated last year
• smonsays / hypernetwork-attention

  View on GitHub
  Official code for the paper "Attention as a Hypernetwork"
  ☆47Jun 22, 2024Updated last year
• jungokasai / T2R

  View on GitHub
  ☆14Nov 20, 2022Updated 3 years ago
• dangxingyu / rnn-icrag

  View on GitHub
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆27Apr 17, 2024Updated last year
• acosharma / elita-transformer

  View on GitHub
  Official Repository for Efficient Linear-Time Attention Transformers.
  ☆18Jun 2, 2024Updated last year
• sustcsonglin / disco-pointer

  View on GitHub
  Official Implementation of ACL2023: Don't Parse, Choose Spans! Continuous and Discontinuous Constituency Parsing via Autoregressive Span …
  ☆14Aug 25, 2023Updated 2 years ago
• VPeterV / RankSpace-Models

  View on GitHub
  source code for NAACL2022 main conference "Dynamic Programming in Rank Space: Scaling Structured Inference with Low-Rank HMMs and PCFGs"
  ☆10Sep 26, 2022Updated 3 years ago
• HazyResearch / embroid

  View on GitHub
  Embroid: Unsupervised Prediction Smoothing Can Improve Few-Shot Classification
  ☆11Aug 12, 2023Updated 2 years ago
• eryk-mazus / sigh

  View on GitHub
  Seamless Voice Interactions with LLMs
  ☆12Oct 28, 2023Updated 2 years ago
• Doraemonzzz / nanoTransNormer

  View on GitHub
  ☆11Oct 11, 2023Updated 2 years ago
• rycolab / aflt-f2023

  View on GitHub
  Advanced Formal Language Theory (263-5352-00L; Frühjahr 2023)
  ☆10Feb 21, 2023Updated 2 years ago
• deep-spin / spectra-rationalization

  View on GitHub
  Repository for SPECTRA: Sparse Structured Text Rationalization, accepted at EMNLP 2021 main conference.
  ☆10Feb 14, 2024Updated 2 years ago
• yikangshen / MoA

  View on GitHub
  Mixture of Attention Heads
  ☆51Oct 10, 2022Updated 3 years ago
• robert-lieck / RBN

  View on GitHub
  Recursive Bayesian Networks
  ☆11May 11, 2025Updated 9 months ago
• nikhilvyas / SOAP_MUON

  View on GitHub
  Combining SOAP and MUON
  ☆19Feb 11, 2025Updated last year
• xuanlinli17 / autoregressive_inference

  View on GitHub
  Code for "Discovering Non-monotonic Autoregressive Orderings with Variational Inference" (paper and code updated from ICLR 2021)
  ☆12Mar 7, 2024Updated last year
• shawntan / scattermoe

  View on GitHub
  Triton-based implementation of Sparse Mixture of Experts.
  ☆265Oct 3, 2025Updated 4 months ago
• Cranial-XIX / longhorn

  View on GitHub
  Official PyTorch Implementation of the Longhorn Deep State Space Model
  ☆56Dec 4, 2024Updated last year
• emorynlp / seq2seq-corenlp

  View on GitHub
  ☆13Feb 7, 2023Updated 3 years ago
• RakitinDen / pytorch-recursive-gumbel-max-trick

  View on GitHub
  Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces, NeurIPS 2021
  ☆14Dec 11, 2021Updated 4 years ago
• OpenNLPLab / ETSC-Exact-Toeplitz-to-SSM-Conversion

  View on GitHub
  [EMNLP 2023] Official implementation of the algorithm ETSC: Exact Toeplitz-to-SSM Conversion our EMNLP 2023 paper - Accelerating Toeplitz…
  ☆14Oct 17, 2023Updated 2 years ago
• iwiwi / epochraft

  View on GitHub
  Checkpointable dataset utilities for foundation model training
  ☆32Jan 29, 2024Updated 2 years ago
• DRSY / KV_Compression

  View on GitHub
  [EMNLP 2023]Context Compression for Auto-regressive Transformers with Sentinel Tokens
  ☆25Nov 6, 2023Updated 2 years ago
• annosubmission / GRC-Cache

  View on GitHub
  ☆16Mar 13, 2023Updated 2 years ago
• OpenSparseLLMs / Linear-MoE

  View on GitHub
  ☆129Jun 6, 2025Updated 8 months ago
• RobertCsordas / moeut

  View on GitHub
  ☆91Aug 18, 2024Updated last year
• zsLin177 / SRL-as-GP

  View on GitHub
  ☆18Mar 10, 2023Updated 2 years ago
• teffland / ner-expected-entity-ratio

  View on GitHub
  Implementation and experiments for Partially Supervised NER via Expected Entity Ratio in TACL 2022
  ☆14Nov 7, 2022Updated 3 years ago
• VITA-Group / Data-Efficient-Scaling

  View on GitHub
  [ICML 2023] "Data Efficient Neural Scaling Law via Model Reusing" by Peihao Wang, Rameswar Panda, Zhangyang Wang
  ☆14Jan 4, 2024Updated 2 years ago
• automl / unlocking_state_tracking

  View on GitHub
  Expanding linear RNN state-transition matrix eigenvalues to include negatives improves state-tracking tasks and language modeling without…
  ☆20Mar 15, 2025Updated 11 months ago
• JeanKaddour / NoTrainNoGain

  View on GitHub
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆81Aug 30, 2023Updated 2 years ago
• shawntan / SUT

  View on GitHub
  Repository for Sparse Universal Transformers
  ☆20Oct 23, 2023Updated 2 years ago