Alternatives and similar repositories for MC-SMoE:

Users that are interested in MC-SMoE are comparing it to the libraries listed below

• hemingkx / TokenSkip
  TokenSkip: Controllable Chain-of-Thought Compression in LLMs
  ☆98Updated last week
• OpenSparseLLMs / Linear-MoE
  ☆70Updated last week
• Lucky-Lance / Expert_Sparsity
  [ACL 2024] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  ☆85Updated 10 months ago
• hahnyuan / ASVD4LLM
  Activation-aware Singular Value Decomposition for Compressing Large Language Models
  ☆59Updated 5 months ago
• CASE-Lab-UMD / Unified-MoE-Compression
  The official implementation of the paper "Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques (TMLR)".
  ☆61Updated this week
• locuslab / massive-activations
  Code accompanying the paper "Massive Activations in Large Language Models"
  ☆150Updated last year
• hemingkx / SWIFT
  [ICLR 2025] SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
  ☆42Updated last month
• hdong920 / GRIFFIN
  ☆36Updated 6 months ago
• GATECH-EIC / ACT
  [ICML 2024] Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrati…
  ☆33Updated 8 months ago
• princeton-nlp / ProLong
  Homepage for ProLong (Princeton long-context language models) and paper "How to Train Long-Context Language Models (Effectively)"
  ☆166Updated 2 weeks ago
• htqin / IR-QLoRA
  [ICML 2024 Oral] This project is the official implementation of our Accurate LoRA-Finetuning Quantization of LLMs via Information Retenti…
  ☆63Updated 11 months ago
• mit-han-lab / x-attention
  XAttention: Block Sparse Attention with Antidiagonal Scoring
  ☆102Updated this week
• hdong920 / LESS
  ☆49Updated 10 months ago
• OpenSparseLLMs / MoM
  ☆71Updated this week
• GCYZSL / MoLA
  ☆127Updated 8 months ago
• SempraETY / Pruning-via-Merging
  ☆17Updated 3 months ago
• horseee / CoT-Valve
  CoT-Valve: Length-Compressible Chain-of-Thought Tuning
  ☆55Updated last month
• machilusZ / FastGen
  This repo contains the source code for: Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
  ☆34Updated 7 months ago
• sramshetty / mixture-of-depths
  An unofficial implementation of "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆34Updated 9 months ago
• FasterDecoding / TEAL
  ☆116Updated last month
• ldery / Bonsai
  Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"
  ☆27Updated 11 months ago
• XunhaoLai / native-sparse-attention-triton
  Efficient triton implementation of Native Sparse Attention.
  ☆116Updated this week
• raymin0223 / fast_robust_early_exit
  Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)
  ☆57Updated 5 months ago
• Shwai-He / MEO
  The source code of "Merging Experts into One: Improving Computational Efficiency of Mixture of Experts (EMNLP 2023)":
  ☆36Updated 11 months ago
• jiwonsong-dev / SLEB
  Official Implementation of SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
  ☆35Updated last month
• mutonix / pyramidinfer
  ☆38Updated 4 months ago
• nbasyl / DoRA
  Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
  ☆123Updated 10 months ago
• fla-org / flame
  🔥 A minimal training framework for scaling FLA models
  ☆82Updated this week
• BaiTheBest / SparseLLM
  Official Repo for SparseLLM: Global Pruning of LLMs (NeurIPS 2024)
  ☆52Updated last month
• Dereck0602 / Awesome_Test_Time_LLMs
  ☆83Updated last week