Alternatives and similar repositories for Skywork-MoE:

Users that are interested in Skywork-MoE are comparing it to the libraries listed below

• YuchuanTian / RethinkTinyLM
  [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”
  ☆121Updated 2 months ago
• IEIT-Yuan / Yuan2.0-M32
  Mixture-of-Experts (MoE) Language Model
  ☆185Updated 6 months ago
• OpenLMLab / scaling-rope
  code for Scaling Laws of RoPE-based Extrapolation
  ☆70Updated last year
• thunlp / Delta-CoMe
  Delta-CoMe can achieve near loss-less 1-bit compressin which has been accepted by NeurIPS 2024
  ☆54Updated 3 months ago
• THUDM / LongAlign
  [EMNLP 2024] LongAlign: A Recipe for Long Context Alignment of LLMs
  ☆241Updated 2 months ago
• GAIR-NLP / ProX
  Offical Repo for "Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale"
  ☆226Updated 3 weeks ago
• GAIR-NLP / ReAlign
  Reformatted Alignment
  ☆114Updated 5 months ago
• 18907305772 / FuseAI
  FuseAI Project
  ☆83Updated last month
• shizhediao / Post-Training-Data-Flywheel
  We aim to provide the best references to search, select, and synthesize high-quality and large-quantity data for post-training your LLMs.
  ☆53Updated 5 months ago
• TemporaryLoRA / Temp-LoRA
  ☆96Updated 11 months ago
• Open-Source-O1 / o1_Reasoning_Patterns_Study
  ☆101Updated 3 months ago
• SomeoneKong / llm_long_context_bench202405
  ☆28Updated 6 months ago
• FlagOpen / Infinity-Instruct
  ☆44Updated 8 months ago
• SparkJiao / llama-pipeline-parallel
  A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to…
  ☆55Updated last year
• FreedomIntelligence / FastLLM
  Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];
  ☆36Updated last year
• jshuadvd / LongRoPE
  Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
  ☆127Updated 7 months ago
• zms1999 / SmartMoE
  A MoE impl for PyTorch, [ATC'23] SmartMoE
  ☆61Updated last year
• jiahe7ay / infini-mini-transformer
  This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…
  ☆56Updated 10 months ago
• HarderThenHarder / RLLoggingBoard
  A visuailzation tool to make deep understaning and easier debugging for RLHF training.
  ☆164Updated 3 weeks ago
• SkyworkAI / skywork-o1-prm-inference
  ☆59Updated 3 months ago
• princeton-nlp / CEPE
  [ACL 2024] Long-Context Language Modeling with Parallel Encodings
  ☆153Updated 9 months ago
• zexuanqiu / CLongEval
  CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
  ☆40Updated last year