Alternatives and similar repositories for ReMoE:

Users that are interested in ReMoE are comparing it to the libraries listed below

• fla-org / flame
  🔥 A minimal training framework for scaling FLA models
  ☆82Updated this week
• OpenSparseLLMs / Linearization
  ☆36Updated this week
• FasterDecoding / TEAL
  ☆116Updated last month
• OpenSparseLLMs / Linear-MoE
  ☆70Updated last week
• sustcsonglin / linear-attention-and-beyond-slides
  ☆63Updated last month
• mit-han-lab / x-attention
  XAttention: Block Sparse Attention with Antidiagonal Scoring
  ☆102Updated this week
• XunhaoLai / native-sparse-attention-triton
  Efficient triton implementation of Native Sparse Attention.
  ☆116Updated this week
• OpenSparseLLMs / MoM
  ☆71Updated this week
• jxiw / MambaInLlama
  [NeurIPS 2024] Official Repository of The Mamba in the Llama: Distilling and Accelerating Hybrid Models
  ☆205Updated 3 weeks ago
• Lucky-Lance / Expert_Sparsity
  [ACL 2024] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  ☆85Updated 10 months ago
• thu-nics / MoA
  The official implementation of the paper <MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression>
  ☆121Updated 3 months ago
• htqin / IR-QLoRA
  [ICML 2024 Oral] This project is the official implementation of our Accurate LoRA-Finetuning Quantization of LLMs via Information Retenti…
  ☆63Updated 11 months ago
• YuchuanTian / DiJiang
  [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…
  ☆99Updated 9 months ago
• NVlabs / COAT
  [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Training
  ☆164Updated last month
• ldery / Bonsai
  Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"
  ☆27Updated 11 months ago
• UNITES-Lab / MC-SMoE
  [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"
  ☆74Updated 9 months ago
• sail-sg / scaling-with-vocab
  [NeurIPS-2024] 📈 Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies https://arxiv.org/abs/2407.13623
  ☆80Updated 5 months ago
• itsnamgyu / block-transformer
  Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
  ☆150Updated 3 months ago
• mit-han-lab / Block-Sparse-Attention
  A sparse attention kernel supporting mix sparse patterns
  ☆168Updated last month
• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆48Updated last week
• SalesforceAIResearch / GemFilter
  ☆76Updated 2 months ago
• hdong920 / GRIFFIN
  ☆36Updated 6 months ago
• RobertCsordas / moeut
  ☆74Updated 7 months ago
• hahnyuan / ASVD4LLM
  Activation-aware Singular Value Decomposition for Compressing Large Language Models
  ☆59Updated 5 months ago
• bytedance / FlexPrefill
  Code for paper: [ICLR2025 Oral] FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
  ☆62Updated this week
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆97Updated 5 months ago
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆154Updated 9 months ago
• sail-sg / SimLayerKV
  The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.
  ☆43Updated 5 months ago
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆125Updated last year
• jiwonsong-dev / SLEB
  Official Implementation of SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
  ☆35Updated last month