Alternatives and similar repositories for Megatron-LM:

Users that are interested in Megatron-LM are comparing it to the libraries listed below

• NVIDIA / FasterTransformer
  Transformer related optimization, including BERT, GPT
  ☆5,981Updated 9 months ago
• Dao-AILab / flash-attention
  Fast and memory-efficient exact attention
  ☆15,064Updated this week
• huggingface / accelerate
  🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (i…
  ☆8,178Updated this week
• huggingface / peft
  🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
  ☆16,978Updated this week
• bitsandbytes-foundation / bitsandbytes
  Accessible large language models via k-bit quantization for PyTorch.
  ☆6,522Updated this week
• huggingface / trl
  Train transformer language models with reinforcement learning.
  ☆10,609Updated this week
• microsoft / DeepSpeed
  DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
  ☆36,255Updated this week
• facebookresearch / fairscale
  PyTorch extensions for high performance and large scale training.
  ☆3,232Updated this week
• google-research / text-to-text-transfer-transformer
  Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"
  ☆6,240Updated 3 months ago
• google / sentencepiece
  Unsupervised text tokenizer for Neural Network-based text generation.
  ☆10,479Updated last month
• facebookresearch / xformers
  Hackable and optimized Transformers building blocks, supporting a composable construction.
  ☆8,910Updated this week
• microsoft / LoRA
  Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"
  ☆11,119Updated last month
• jessevig / bertviz
  BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)
  ☆7,075Updated last year
• microsoft / unilm
  Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
  ☆20,584Updated last week
• artidoro / qlora
  QLoRA: Efficient Finetuning of Quantized LLMs
  ☆10,168Updated 7 months ago
• triton-lang / triton
  Development repository for the Triton language and compiler
  ☆14,042Updated this week
• google-research / t5x
  ☆2,723Updated this week
• facebookresearch / fairseq
  Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
  ☆30,809Updated last week
• EleutherAI / lm-evaluation-harness
  A framework for few-shot evaluation of language models.
  ☆7,474Updated this week
• BlinkDL / RWKV-LM
  RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable)…
  ☆13,005Updated last week
• microsoft / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,957Updated 3 weeks ago
• NVIDIA / NeMo
  A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Auto…
  ☆12,864Updated this week
• microsoft / DeepSpeedExamples
  Example models using DeepSpeed
  ☆6,220Updated last week
• UKPLab / sentence-transformers
  State-of-the-Art Text Embeddings
  ☆15,772Updated last week
• EleutherAI / gpt-neox
  An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries
  ☆7,042Updated this week
• vllm-project / vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆33,809Updated this week
• NVIDIA / apex
  A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch
  ☆8,502Updated last month
• alpa-projects / alpa
  Training and serving large-scale neural networks with auto parallelization.
  ☆3,093Updated last year
• CarperAI / trlx
  A repo for distributed training of language models with Reinforcement Learning via Human Feedback (RLHF)
  ☆4,567Updated last year
• NVIDIA / TensorRT-LLM
  TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain…
  ☆9,147Updated this week