Alternatives and similar repositories for distributed-training-guide:

Users that are interested in distributed-training-guide are comparing it to the libraries listed below

• huggingface / picotron
  Minimalistic 4D-parallelism distributed training framework for education purpose
  ☆722Updated last week
• huggingface / picotron_tutorial
  ☆133Updated this week
• facebookresearch / spdl
  Scalable and Performant Data Loading
  ☆217Updated last week
• facebookresearch / memory
  Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…
  ☆294Updated 2 months ago
• EleutherAI / cookbook
  Deep learning for dummies. All the practical details and useful utilities that go into working with real models.
  ☆770Updated 3 weeks ago
• facebookresearch / optimizers
  For optimization algorithm research and development.
  ☆490Updated last week
• open-thought / system-2-research
  System 2 Reasoning Link Collection
  ☆791Updated last week
• foundation-model-stack / fms-fsdp
  🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…
  ☆221Updated this week
• rkinas / triton-resources
  A curated list of resources for learning and exploring Triton, OpenAI's programming language for writing efficient GPU code.
  ☆265Updated this week
• facebookresearch / LayerSkip
  Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024
  ☆270Updated this week
• facebookresearch / coconut
  Training Large Language Model to Reason in a Continuous Latent Space
  ☆840Updated 3 weeks ago
• rwitten / HighPerfLLMs2024
  ☆359Updated 7 months ago
• pytorch / torchft
  PyTorch per step fault tolerance (actively under development)
  ☆243Updated this week
• jxmorris12 / cde
  code for training & evaluating Contextual Document Embedding models
  ☆173Updated last month
• BobMcDear / attorch
  A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.
  ☆514Updated this week
• huggingface / search-and-learn
  Recipes to scale inference-time compute of open models
  ☆993Updated last month
• NVIDIA / kvpress
  LLM KV cache compression made easy
  ☆389Updated this week
• gpu-mode / profiling-cuda-in-torch
  ☆141Updated last year
• clu0 / unet.cu
  UNet diffusion model in pure CUDA
  ☆598Updated 7 months ago
• KellerJordan / Muon
  Muon optimizer: +~30% sample efficiency with <3% wallclock overhead
  ☆252Updated last week
• NVIDIA / ngpt
  Normalized Transformer (nGPT)
  ☆150Updated 2 months ago
• pytorch-labs / attention-gym
  Helpful tools and examples for working with flex-attention
  ☆631Updated this week
• predibase / llm_distillation_playbook
  Best practices for distilling large language models.
  ☆476Updated last year
• MarioSieg / magnetron
  (WIP) A small but powerful, homemade PyTorch from scratch.
  ☆522Updated this week
• srush / annotated-mamba
  Annotated version of the Mamba paper
  ☆472Updated 11 months ago
• microsoft / Samba
  Official implementation of "Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling"
  ☆840Updated 3 weeks ago
• hkproj / triton-flash-attention
  ☆126Updated last month
• huggingface / nanotron
  Minimalistic large language model 3D-parallelism training
  ☆1,451Updated this week