Alternatives and similar repositories for Megatron-LM

Users that are interested in Megatron-LM are comparing it to the libraries listed below

• CoinCheung / gdGPT
  Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.
  ☆98Updated last year
• zms1999 / SmartMoE
  A MoE impl for PyTorch, [ATC'23] SmartMoE
  ☆71Updated 2 years ago
• Glaciohound / LM-Infinite
  Implementation of NAACL 2024 Outstanding Paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"
  ☆152Updated 10 months ago
• HuangLK / transpeeder
  train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism
  ☆224Updated 2 years ago
• hpcaitech / PaLM-colossalai
  Scalable PaLM implementation of PyTorch
  ☆190Updated 3 years ago
• jiahe7ay / infini-mini-transformer
  This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…
  ☆58Updated last year
• MoFHeka / LLaMA-Megatron
  A LLaMA1/LLaMA12 Megatron implement.
  ☆28Updated 2 years ago
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆139Updated last year
• MayDomine / Burst-Attention
  Distributed IO-aware Attention algorithm
  ☆23Updated 4 months ago
• kyegomez / FlashAttention20Triton
  Triton implementation of Flash Attention2.0
  ☆47Updated 2 years ago
• FreedomIntelligence / FastLLM
  Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];
  ☆40Updated 2 years ago
• ProjectD-AI / LLaMA-Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆69Updated 2 years ago
• bojone / rerope
  Rectified Rotary Position Embeddings
  ☆387Updated last year
• Ascend / AscendSpeed
  ☆79Updated 2 years ago
• genggui001 / Megatron-DeepSpeed-Llama
  ☆84Updated 2 years ago
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆113Updated 10 months ago
• OpenNLPLab / LASP
  Linear Attention Sequence Parallelism (LASP)
  ☆88Updated last year
• DAMO-NLP-SG / CLEX
  [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models
  ☆78Updated last year
• OpenBMB / infllmv2_cuda_impl
  ☆86Updated last week
• liyucheng09 / llm-compressive
  Longitudinal Evaluation of LLMs via Data Compression
  ☆33Updated last year
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆137Updated 2 years ago
• THUDM / LongAlign
  [EMNLP 2024] LongAlign: A Recipe for Long Context Alignment of LLMs
  ☆260Updated last year
• yegcjs / mixinglaws
  ☆109Updated 6 months ago
• nick7nlp / Counting-Stars
  Counting-Stars (★)
  ☆83Updated 2 months ago
• beichao1314 / Open-Llama
  The complete training code of the open-source high-performance Llama model, including the full process from pre-training to RLHF.
  ☆67Updated 2 years ago
• OpenLMLab / scaling-rope
  code for Scaling Laws of RoPE-based Extrapolation
  ☆73Updated 2 years ago
• qhliu26 / Dive-into-Big-Model-Training
  📑 Dive into Big Model Training
  ☆116Updated 3 years ago
• haochengxi / Train_Transformers_with_INT4
  ☆157Updated 2 years ago
• hpcaitech / Elixir
  Elixir: Train a Large Language Model on a Small GPU Cluster
  ☆15Updated 2 years ago
• FasterDecoding / REST
  REST: Retrieval-Based Speculative Decoding, NAACL 2024
  ☆214Updated 4 months ago