Alternatives and similar repositories for AutoMoE:

Users that are interested in AutoMoE are comparing it to the libraries listed below

• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆51Updated 2 years ago
• HazyResearch / prefix-linear-attention
  ☆53Updated 10 months ago
• nverma1 / merging-text-transformers
  Code for "Merging Text Transformers from Different Initializations"
  ☆20Updated 3 months ago
• kyegomez / Blockwise-Parallel-Transformer
  32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.
  ☆48Updated last year
• rayleizhu / vllm-ra
  [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts
  ☆39Updated last year
• kamanphoebe / Look-into-MoEs
  [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models
  ☆51Updated 3 months ago
• Infini-AI-Lab / gsm_infinite
  ☆45Updated 2 months ago
• Arenaa / Accelerated-Generation-Techniques
  This repository contains papers for a comprehensive survey on accelerated generation techniques in Large Language Models (LLMs).
  ☆11Updated 11 months ago
• JeanKaddour / NoTrainNoGain
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆80Updated last year
• microsoft / Stochastic-Mixture-of-Experts
  This package implements THOR: Transformer with Stochastic Experts.
  ☆61Updated 3 years ago
• thunlp / Modularity-Analysis
  Repo for ACL2023 Findings paper "Emergent Modularity in Pre-trained Transformers"
  ☆23Updated last year
• IST-DASLab / SparseFinetuning
  Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry
  ☆40Updated last year
• PiotrNawrot / sparse-frontier
  Official implementation of "The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs"
  ☆26Updated 2 weeks ago
• daochenzha / dreamshard
  [NeurIPS 2022] DreamShard: Generalizable Embedding Table Placement for Recommender Systems
  ☆29Updated 2 years ago
• prateeky2806 / ComPEFT
  ☆25Updated last year
• IST-DASLab / QIGen
  Repository for CPU Kernel Generation for LLM Inference
  ☆26Updated last year
• ldery / Bonsai
  Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"
  ☆28Updated last year
• juvi21 / CoPE-cuda
  Contextual Position Encoding but with some custom CUDA Kernels https://arxiv.org/abs/2405.18719
  ☆22Updated 11 months ago
• OpenNLPLab / LASP
  Linear Attention Sequence Parallelism (LASP)
  ☆82Updated 11 months ago
• giangdip2410 / HyperRouter
  Code for this paper "HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts via HyperNetwork"
  ☆31Updated last year
• aniquetahir / JORA
  JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)
  ☆33Updated last year
• yxli2123 / LoSparse
  ☆51Updated last year
• pprp / Pruner-Zero
  Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMs
  ☆81Updated 5 months ago
• GATECH-EIC / Linearized-LLM
  [ICML 2024] When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
  ☆30Updated 10 months ago
• VITA-Group / Junk_DNA_Hypothesis
  [ICML 2024] Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity; Lu Yin*, Ajay Jaiswal*, Shiwei Liu, So…
  ☆16Updated 2 weeks ago
• shreyansh26 / Attention-Mask-Patterns
  Using FlexAttention to compute attention with different masking patterns
  ☆43Updated 7 months ago
• Hunter-DDM / stablemoe
  Code for the ACL-2022 paper "StableMoE: Stable Routing Strategy for Mixture of Experts"
  ☆45Updated 2 years ago
• locuslab / scaling_laws_data_filtering
  ☆64Updated last year
• hdong920 / GRIFFIN
  ☆37Updated 8 months ago
• dangxingyu / rnn-icrag
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆27Updated last year