Alternatives and similar repositories for picotron_tutorial:

Users that are interested in picotron_tutorial are comparing it to the libraries listed below

• cloneofsimo / min-fsdp
  ☆75Updated 6 months ago
• facebookresearch / memory
  Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…
  ☆254Updated 3 weeks ago
• SeunghyunSEO / optimized_hf_llama_class_for_training
  ☆47Updated 4 months ago
• EleutherAI / nanoGPT-mup
  The simplest, fastest repository for training/finetuning medium-sized GPTs.
  ☆89Updated last month
• foundation-model-stack / bamba
  Train, tune, and infer Bamba model
  ☆73Updated this week
• joey00072 / ohara
  Collection of autoregressive model implementation
  ☆76Updated this week
• gpu-mode / profiling-cuda-in-torch
  ☆138Updated 11 months ago
• rasbt / pytorch-memory-optim
  This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…
  ☆87Updated last year
• leloykun / modded-nanogpt
  NanoGPT (124M) quality in 2.67B tokens
  ☆24Updated this week
• Zyphra / tree_attention
  Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
  ☆109Updated last month
• gpu-mode / ring-attention
  ring-attention experiments
  ☆113Updated 2 months ago
• foundation-model-stack / fms-fsdp
  🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…
  ☆213Updated this week
• cloneofsimo / min-max-gpt
  Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training
  ☆121Updated 8 months ago
• warner-benjamin / optimi
  Fast, Modern, Memory Efficient, and Low Precision PyTorch Optimizers
  ☆77Updated 5 months ago
• stas00 / ml-ways
  ML/DL Math and Method notes
  ☆57Updated last year
• xrsrke / pipegoose
  Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
  ☆81Updated last year
• google-deepmind / asyncdiloco
  ☆40Updated 11 months ago
• huggingface / chug
  Minimal sharded dataset loaders, decoders, and utils for multi-modal document, image, and text datasets.
  ☆155Updated 9 months ago
• epfml / DenseFormer
  ☆78Updated 8 months ago
• siyan-zhao / prepacking
  The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"
  ☆57Updated 3 months ago
• melisa-writer / short-transformers
  Prune transformer layers
  ☆67Updated 7 months ago
• pytorch / torchft
  PyTorch per step fault tolerance (actively under development)
  ☆209Updated this week
• LambdaLabsML / distributed-training-guide
  Best practices & guides on how to write distributed pytorch training code
  ☆329Updated 3 weeks ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 7 months ago
• luyug / magix
  Supercharge huggingface transformers with model parallelism.
  ☆75Updated 3 months ago
• apple / ml-sigma-reparam
  ☆296Updated 6 months ago
• facebookresearch / spdl
  Scalable and Performant Data Loading
  ☆201Updated this week
• softmax1 / Flash-Attention-Softmax-N
  CUDA and Triton implementations of Flash Attention with SoftmaxN.
  ☆67Updated 7 months ago
• huggingface / optimum-tpu
  Google TPU optimizations for transformers models
  ☆84Updated this week
• lucidrains / PEER-pytorch
  Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind
  ☆115Updated 4 months ago