Alternatives and similar repositories for Megatron-LM

Users that are interested in Megatron-LM are comparing it to the libraries listed below

• Azure / msccl
  Microsoft Collective Communication Library
  ☆66Updated 10 months ago
• intel / intel-extension-for-deepspeed
  Intel® Extension for DeepSpeed* is an extension to DeepSpeed that brings feature support with SYCL kernels on Intel GPU(XPU) device. Note…
  ☆63Updated 3 months ago
• ROCm / MAD
  MAD (Model Automation and Dashboarding)
  ☆25Updated last week
• ROCm / TransformerEngine
  ☆48Updated this week
• triton-lang / kernels
  ☆92Updated 11 months ago
• HabanaAI / vllm-fork
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆83Updated this week
• ROCm / vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆108Updated this week
• mk1-project / quickreduce
  QuickReduce is a performant all-reduce library designed for AMD ROCm that supports inline compression.
  ☆33Updated last month
• microsoft / nnscaler
  nnScaler: Compiling DNN models for Parallel Training
  ☆118Updated 3 weeks ago
• yifuwang / symm-mem-recipes
  ☆124Updated 9 months ago
• usyd-fsalab / fp6_llm
  An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).
  ☆265Updated 3 months ago
• ROCm / rccl-tests
  RCCL Performance Benchmark Tests
  ☆75Updated last week
• ROCm / iris
  AMD RAD's multi-GPU Triton-based framework for seamless multi-GPU programming
  ☆83Updated last week
• hao-ai-lab / MuxServe
  ☆72Updated last year
• AlibabaResearch / flash-llm
  Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity
  ☆221Updated 2 years ago
• Azure / msccl-executor-nccl
  ☆46Updated 10 months ago
• tgale96 / grouped_gemm
  PyTorch bindings for CUTLASS grouped GEMM.
  ☆124Updated 4 months ago
• ROCm / triton
  Development repository for the Triton language and compiler
  ☆135Updated this week
• ROCm / aiter
  AI Tensor Engine for ROCm
  ☆285Updated last week
• RulinShao / LightSeq
  Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training
  ☆216Updated last year
• ROCm / flash-attention
  Fast and memory-efficient exact attention
  ☆193Updated last week
• ParCIS / Chimera
  Chimera: bidirectional pipeline parallelism for efficiently training large-scale models.
  ☆66Updated 7 months ago
• infinigence / FlashOverlap
  A lightweight design for computation-communication overlap.
  ☆181Updated last week
• intel / torch-ccl
  oneCCL Bindings for Pytorch*
  ☆102Updated 2 months ago
• AMD-AGI / Primus-Turbo
  ☆25Updated this week
• mcrl / tccl
  Thunder Research Group's Collective Communication Library
  ☆42Updated 3 months ago
• meta-pytorch / applied-ai
  Applied AI experiments and examples for PyTorch
  ☆299Updated last month
• DachengLi1 / AMP
  (NeurIPS 2022) Automatically finding good model-parallel strategies, especially for complex models and clusters.
  ☆41Updated 2 years ago
• flexflow / flexflow-serve
  FlexFlow Serve: Low-Latency, High-Performance LLM Serving
  ☆63Updated last month
• zhuohan123 / terapipe
  ☆75Updated 4 years ago