Alternatives and similar repositories for DistributedTrainingGPT2

Users that are interested in DistributedTrainingGPT2 are comparing it to the libraries listed below

• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆110Updated 6 months ago
• NVIDIA-NeMo / Megatron-Bridge
  Training library for Megatron-based models
  ☆85Updated this week
• RulinShao / LightSeq
  Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training
  ☆216Updated last year
• zms1999 / SmartMoE
  A MoE impl for PyTorch, [ATC'23] SmartMoE
  ☆70Updated 2 years ago
• MoFHeka / LLaMA-Megatron
  A LLaMA1/LLaMA12 Megatron implement.
  ☆28Updated last year
• CoinCheung / gdGPT
  Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.
  ☆98Updated last year
• cofe-ai / Mu-scaling
  Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales
  ☆32Updated 2 years ago
• infinigence / LVEval
  Repository of LV-Eval Benchmark
  ☆70Updated last year
• sail-sg / VocabularyParallelism
  Vocabulary Parallelism
  ☆22Updated 6 months ago
• ISEEKYAN / mbridge
  Bridge Megatron-Core to Hugging Face/Reinforcement Learning
  ☆126Updated this week
• FasterDecoding / REST
  REST: Retrieval-Based Speculative Decoding, NAACL 2024
  ☆208Updated last week
• Equationliu / Kangaroo
  [NeurIPS 2024] The official implementation of "Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exitin…
  ☆60Updated last year
• Jikai0Wang / OPT-Tree
  ☆27Updated 3 months ago
• NJUNLP / MCSD
  Multi-Candidate Speculative Decoding
  ☆36Updated last year
• ISEEKYAN / verl_megatron_practice
  (best/better) practices of megatron on veRL and tuning guide
  ☆86Updated 3 weeks ago
• Dao-AILab / grouped-latent-attention
  ☆126Updated 3 months ago
• FasterDecoding / SnapKV
  ☆279Updated 2 months ago
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆131Updated last year
• FFY0 / AdaKV
  The Official Implementation of Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  ☆92Updated 3 months ago
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆127Updated 9 months ago
• sail-sg / SimLayerKV
  The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.
  ☆48Updated 11 months ago
• Gaffey / ExCP
  Official implementation of ICML 2024 paper "ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking".
  ☆48Updated last year
• dilab-zju / self-speculative-decoding
  Code associated with the paper **Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding**
  ☆202Updated 7 months ago
• yaof20 / Flash-RL
  Implementation for FP8/INT8 Rollout for RL training without performence drop.
  ☆207Updated last week
• thu-pacman / SmartMoE-AE
  ATC23 AE
  ☆46Updated 2 years ago
• feifeibear / Odysseus-Transformer
  Odysseus: Playground of LLM Sequence Parallelism
  ☆77Updated last year
• CalvinXKY / mfu_calculation
  A simple calculation for LLM MFU.
  ☆44Updated last week
• opengear-project / GEAR
  GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM
  ☆168Updated last year
• machilusZ / FastGen
  This repo contains the source code for: Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
  ☆39Updated last year
• Ascend / AscendSpeed
  ☆79Updated last year