Alternatives and similar repositories for llm-surgeon:

Users that are interested in llm-surgeon are comparing it to the libraries listed below

• JeanKaddour / NoTrainNoGain
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆79Updated last year
• sramshetty / mixture-of-depths
  An unofficial implementation of "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆35Updated 8 months ago
• locuslab / massive-activations
  Code accompanying the paper "Massive Activations in Large Language Models"
  ☆140Updated 11 months ago
• PiotrNawrot / nano-sparse-attention
  The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.
  ☆57Updated 3 weeks ago
• kyleliang919 / Online-Subspace-Descent
  This repo is based on https://github.com/jiaweizzhao/GaLore
  ☆24Updated 5 months ago
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆70Updated 3 months ago
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆48Updated last year
• jiwonsong-dev / SLEB
  Official Implementation of SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
  ☆33Updated 2 weeks ago
• JonasGeiping / linear_cross_entropy_loss
  A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.
  ☆62Updated 6 months ago
• IST-DASLab / RoSA
  Official implementation of the ICML 2024 paper RoSA (Robust Adaptation)
  ☆38Updated last year
• RobertCsordas / moeut
  ☆71Updated 6 months ago
• fkodom / soft-mixture-of-experts
  PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
  ☆71Updated last year
• ScalingIntelligence / large_language_monkeys
  ☆82Updated 4 months ago
• IST-DASLab / SparseFinetuning
  Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry
  ☆40Updated last year
• mengxiayu / LLMSuperWeight
  Code for studying the super weight in LLM
  ☆80Updated 2 months ago
• Lucky-Lance / Expert_Sparsity
  [ACL 2024] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  ☆82Updated 8 months ago
• mgmalek / efficient_cross_entropy
  ☆91Updated 8 months ago
• dkopi / Bitune
  Implementation of Bitune: Bidirectional Instruction-Tuning
  ☆19Updated 8 months ago
• JeanKaddour / LAWA
  Latest Weight Averaging (NeurIPS HITY 2022)
  ☆28Updated last year
• hahnyuan / ASVD4LLM
  Activation-aware Singular Value Decomposition for Compressing Large Language Models
  ☆56Updated 3 months ago
• raymin0223 / fast_robust_early_exit
  Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)
  ☆56Updated 4 months ago
• VILA-Lab / GBLM-Pruner
  Are gradient information useful for pruning of LLMs?
  ☆42Updated 9 months ago
• shikaiqiu / compute-better-spent
  ☆52Updated 4 months ago
• VijayLingam95 / SVFT
  ☆26Updated last week
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆96Updated 4 months ago
• siyan-zhao / prepacking
  The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"
  ☆59Updated 4 months ago
• HazyResearch / prefix-linear-attention
  ☆49Updated 7 months ago
• dmis-lab / Monet
  [ICLR 2025] Monet: Mixture of Monosemantic Experts for Transformers
  ☆56Updated 3 weeks ago
• UNITES-Lab / MC-SMoE
  [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"
  ☆72Updated 8 months ago
• epfml / dynamic-sparse-flash-attention
  ☆138Updated last year