Alternatives and similar repositories for nanoGPT-moe:

Users that are interested in nanoGPT-moe are comparing it to the libraries listed below

• schwartz-lab-NLP / TOVA
  Token Omission Via Attention
  ☆123Updated 4 months ago
• Zyphra / tree_attention
  Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
  ☆116Updated 2 months ago
• minyoungg / LTE
  ☆66Updated 7 months ago
• wdlctc / mini-s
  ☆44Updated 3 months ago
• kanishkg / stream-of-search
  Repository for the paper Stream of Search: Learning to Search in Language
  ☆138Updated 2 weeks ago
• OswaldHe / HMT-pytorch
  [NAACL 2025] Official Implementation of "HMT: Hierarchical Memory Transformer for Long Context Language Processing"
  ☆67Updated 2 weeks ago
• BlinkDL / modded-nanogpt-rwkv
  RWKV-7: Surpassing GPT
  ☆79Updated 3 months ago
• xrsrke / pipegoose
  Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
  ☆81Updated last year
• cmu-l3 / neurips2024-inference-tutorial-code
  NeurIPS 2024 tutorial on LLM Inference
  ☆39Updated 2 months ago
• HanGuo97 / lq-lora
  ☆125Updated last year
• akjindal53244 / Arithmo
  Small and Efficient Mathematical Reasoning LLMs
  ☆71Updated last year
• kernelmachine / cbtm
  Code repository for the c-BTM paper
  ☆105Updated last year
• sanyalsunny111 / LLM-Inheritune
  This is the official repository for Inheritune.
  ☆109Updated last week
• RobertCsordas / moeut
  ☆71Updated 6 months ago
• MNoorFawi / curlora
  The code repository for the CURLoRA research paper. Stable LLM continual fine-tuning and catastrophic forgetting mitigation.
  ☆41Updated 5 months ago
• whyNLP / LCKV
  Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…
  ☆148Updated last month
• KaiNylund / lm-weights-encode-time
  ☆67Updated 6 months ago
• llm-random / llm-random
  ☆181Updated this week
• tanyuqian / redco
  NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference
  ☆64Updated 2 months ago
• JHU-CLSP / RATIONALYST
  Code for RATIONALYST: Pre-training Process-Supervision for Improving Reasoning https://arxiv.org/pdf/2410.01044
  ☆32Updated 4 months ago
• BorealisAI / flora-opt
  This is the official repository for the paper "Flora: Low-Rank Adapters Are Secretly Gradient Compressors" in ICML 2024.
  ☆98Updated 7 months ago
• CASE-Lab-UMD / LLM-Drop
  The official implementation of the paper "What Matters in Transformers? Not All Attention is Needed".
  ☆160Updated 2 months ago
• LucasPrietoAl / grokking-at-the-edge-of-numerical-stability
  ☆84Updated last month
• cloneofsimo / min-max-gpt
  Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training
  ☆122Updated 10 months ago
• AnswerDotAI / cold-compress
  Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of…
  ☆117Updated 6 months ago
• kyegomez / Lets-Verify-Step-by-Step
  "Improving Mathematical Reasoning with Process Supervision" by OPENAI
  ☆103Updated last week
• architsharma97 / dpo-rlaif
  ☆95Updated 7 months ago
• amudide / switch_sae
  Efficient Dictionary Learning with Switch Sparse Autoencoders (SAEs)
  ☆20Updated 2 months ago
• JacobPfau / fillerTokens
  ☆58Updated 9 months ago