Alternatives and similar repositories for Megatron-LM

Users that are interested in Megatron-LM are comparing it to the libraries listed below

• Azure / msccl
  Microsoft Collective Communication Library
  ☆65Updated 8 months ago
• intel / intel-extension-for-deepspeed
  Intel® Extension for DeepSpeed* is an extension to DeepSpeed that brings feature support with SYCL kernels on Intel GPU(XPU) device. Note…
  ☆61Updated last month
• ROCm / TransformerEngine
  ☆41Updated this week
• ROCm / vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆86Updated last week
• triton-lang / kernels
  ☆85Updated 9 months ago
• ROCm / flash-attention
  Fast and memory-efficient exact attention
  ☆179Updated last week
• ROCm / MAD
  MAD (Model Automation and Dashboarding)
  ☆23Updated last week
• ROCm / rccl-tests
  RCCL Performance Benchmark Tests
  ☆71Updated last week
• ROCm / triton
  Development repository for the Triton language and compiler
  ☆127Updated this week
• intel / cutlass-sycl
  A CUTLASS implementation using SYCL
  ☆32Updated this week
• HabanaAI / vllm-fork
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆78Updated this week
• intel / torch-ccl
  oneCCL Bindings for Pytorch*
  ☆99Updated this week
• cchan / tccl
  extensible collectives library in triton
  ☆88Updated 4 months ago
• feifeibear / DPSKV3MFU
  Estimate MFU for DeepSeekV3
  ☆25Updated 7 months ago
• ROCm / aiter
  AI Tensor Engine for ROCm
  ☆243Updated this week
• usyd-fsalab / fp6_llm
  An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).
  ☆260Updated 3 weeks ago
• hao-ai-lab / MuxServe
  ☆67Updated last year
• yifuwang / symm-mem-recipes
  ☆102Updated 7 months ago
• microsoft / nnscaler
  nnScaler: Compiling DNN models for Parallel Training
  ☆114Updated this week
• DachengLi1 / AMP
  (NeurIPS 2022) Automatically finding good model-parallel strategies, especially for complex models and clusters.
  ☆40Updated 2 years ago
• Infrawaves / DeepEP_ibrc_dual-ports_multiQP
  Aims to implement dual-port and multi-qp solutions in deepEP ibrc transport
  ☆58Updated 3 months ago
• tgale96 / grouped_gemm
  PyTorch bindings for CUTLASS grouped GEMM.
  ☆107Updated 2 months ago
• RulinShao / LightSeq
  Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training
  ☆212Updated 11 months ago
• argonne-lcf / LLM-Inference-Bench
  LLM-Inference-Bench
  ☆48Updated 3 weeks ago
• pytorch-labs / applied-ai
  Applied AI experiments and examples for PyTorch
  ☆289Updated 2 months ago
• ROCm / pytorch-micro-benchmarking
  ☆20Updated last week
• infinigence / FlashOverlap
  A lightweight design for computation-communication overlap.
  ☆155Updated last month
• mk1-project / quickreduce
  QuickReduce is a performant all-reduce library designed for AMD ROCm that supports inline compression.
  ☆31Updated 4 months ago
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆123Updated 8 months ago
• mlcommons / training_results_v2.0
  This repository contains the results and code for the MLPerf™ Training v2.0 benchmark.
  ☆29Updated last year