Alternatives and similar repositories for Megatron-Energon

Users that are interested in Megatron-Energon are comparing it to the libraries listed below

• foundation-model-stack / fms-fsdp
  🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…
  ☆258Updated 3 weeks ago
• NVlabs / COAT
  [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Training
  ☆222Updated last week
• fla-org / flame
  🔥 A minimal training framework for scaling FLA models
  ☆224Updated 2 months ago
• RulinShao / LightSeq
  Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training
  ☆213Updated 11 months ago
• lucidrains / ring-attention-pytorch
  Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
  ☆536Updated 3 months ago
• NVIDIA-NeMo / RL
  Scalable toolkit for efficient model reinforcement
  ☆607Updated this week
• alexzhang13 / flashattention2-custom-mask
  Triton implementation of FlashAttention2 that adds Custom Masks.
  ☆132Updated last year
• haoliuhl / ringattention
  Large Context Attention
  ☆724Updated 6 months ago
• ByteDance-Seed / ByteCheckpoint
  ByteCheckpoint: An Unified Checkpointing Library for LFMs
  ☆234Updated last month
• zhuzilin / ring-flash-attention
  Ring attention implementation with flash attention
  ☆836Updated last week
• meta-pytorch / float8_experimental
  This repository contains the experimental PyTorch native float8 training UX
  ☆224Updated last year
• mit-han-lab / duo-attention
  [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
  ☆488Updated 6 months ago
• foundation-model-stack / foundation-model-stack
  🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.
  ☆207Updated this week
• XunhaoLai / native-sparse-attention-triton
  Efficient triton implementation of Native Sparse Attention.
  ☆207Updated 2 months ago
• NVIDIA / Star-Attention
  Efficient LLM Inference over Long Sequences
  ☆387Updated last month
• Dao-AILab / grouped-latent-attention
  ☆123Updated 2 months ago
• gpu-mode / ring-attention
  ring-attention experiments
  ☆149Updated 9 months ago
• nil0x9 / flash-muon
  Flash-Muon: An Efficient Implementation of Muon Optimizer
  ☆157Updated 2 months ago
• Azure / MS-AMP
  Microsoft Automatic Mixed Precision Library
  ☆616Updated 10 months ago
• NVlabs / Fast-dLLM
  Official implementation of "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding"
  ☆346Updated 2 weeks ago
• mit-han-lab / x-attention
  [ICML 2025] XAttention: Block Sparse Attention with Antidiagonal Scoring
  ☆218Updated last month
• sii-research / siiRL
  siiRL: Shanghai Innovation Institute RL Framework for Advanced LLMs and Multi-Agent Systems
  ☆160Updated this week
• shawntan / scattermoe
  Triton-based implementation of Sparse Mixture of Experts.
  ☆233Updated 8 months ago
• feifeibear / long-context-attention
  USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
  ☆546Updated 3 weeks ago
• OpenNLPLab / lightning-attention
  Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  ☆324Updated 5 months ago
• feifeibear / Odysseus-Transformer
  Odysseus: Playground of LLM Sequence Parallelism
  ☆76Updated last year
• huggingface / kernels
  Load compute kernels from the Hub
  ☆237Updated this week
• huggingface / picotron_tutorial
  ☆209Updated 6 months ago
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆130Updated last year
• apple / ml-cross-entropy
  ☆512Updated 2 weeks ago