Alternatives and similar repositories for SparseMixer:

Users that are interested in SparseMixer are comparing it to the libraries listed below

• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆48Updated this week
• ScalingIntelligence / large_language_monkeys
  ☆87Updated 5 months ago
• HazyResearch / prefix-linear-attention
  ☆52Updated 8 months ago
• teelinsan / parallel-decoding
  Repository of the paper "Accelerating Transformer Inference for Translation via Parallel Decoding"
  ☆114Updated last year
• sjelassi / transformers_ssm_copy
  ☆30Updated last year
• berlino / gated_linear_attention
  ☆101Updated last year
• yegcjs / DiffusionLLM
  Code for paper "Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning"
  ☆70Updated last year
• gregorbachmann / Next-Token-Failures
  ☆81Updated last year
• mlfoundations / scaling
  Language models scale reliably with over-training and on downstream tasks
  ☆96Updated 11 months ago
• ldery / Bonsai
  Code for "Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes"
  ☆27Updated 11 months ago
• sail-sg / scaling-with-vocab
  [NeurIPS-2024] 📈 Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies https://arxiv.org/abs/2407.13623
  ☆80Updated 5 months ago
• PKU-ML / LongPPL
  Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"
  ☆44Updated 3 weeks ago
• ryoungj / ObsScaling
  [NeurIPS'24 Spotlight] Observational Scaling Laws
  ☆53Updated 5 months ago
• berlino / seq_icl
  ☆51Updated 9 months ago
• chijames / KERPLE
  ☆19Updated 2 years ago
• thunlp / Modularity-Analysis
  Repo for ACL2023 Findings paper "Emergent Modularity in Pre-trained Transformers"
  ☆23Updated last year
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆48Updated 2 years ago
• sustcsonglin / mamba-triton
  ☆47Updated last year
• kyegomez / Blockwise-Parallel-Transformer
  32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.
  ☆46Updated last year
• raymin0223 / fast_robust_early_exit
  Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)
  ☆56Updated 5 months ago
• Edward-Sun / gpt-accelera
  Simple and efficient pytorch-native transformer training and inference (batched)
  ☆70Updated 11 months ago
• abhishekpanigrahi1996 / transformer_in_transformer
  ☆44Updated last year
• SalesforceAIResearch / GemFilter
  ☆76Updated 2 months ago
• DAMO-NLP-SG / CLEX
  [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models
  ☆76Updated last year
• Shwai-He / MEO
  The source code of "Merging Experts into One: Improving Computational Efficiency of Mixture of Experts (EMNLP 2023)":
  ☆36Updated 11 months ago
• RobertCsordas / moe
  Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"
  ☆36Updated last year