Alternatives and similar repositories for large-model-parallelism:

Users that are interested in large-model-parallelism are comparing it to the libraries listed below

• cloneofsimo / min-max-gpt
  Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training
  ☆121Updated 9 months ago
• AblateIt / finetune-study
  Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.
  ☆82Updated last year
• sangmichaelxie / cs324_p2
  Project 2 (Building Large Language Models) for Stanford CS324: Understanding and Developing Large Language Models (Winter 2022)
  ☆102Updated last year
• NousResearch / StripedHyenaTrainer
  ☆60Updated last year
• ayaka14732 / llama-2-jax
  JAX implementation of the Llama 2 model
  ☆213Updated 11 months ago
• Sea-Snell / JAXSeq
  Train very large language models in Jax.
  ☆198Updated last year
• cloneofsimo / min-fsdp
  ☆75Updated 6 months ago
• sholtodouglas / scalingExperiments
  ☆58Updated 2 years ago
• xrsrke / pipegoose
  Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
  ☆81Updated last year
• srush / GPTWorld
  A puzzle to learn about prompting
  ☆123Updated last year
• google-deepmind / mishax
  ☆118Updated last week
• callummcdougall / sae_visualizer
  ☆25Updated 9 months ago
• kevinwu23 / StanfordFineTuneBench
  ☆27Updated 2 months ago
• EleutherAI / nanoGPT-mup
  The simplest, fastest repository for training/finetuning medium-sized GPTs.
  ☆91Updated 2 months ago
• Alignment-Lab-AI / datagen
  a pipeline for using api calls to agnostically convert unstructured data into structured training data
  ☆29Updated 4 months ago
• epfml / llm-baselines
  nanoGPT-like codebase for LLM training
  ☆85Updated this week
• EleutherAI / semantic-memorization
  ☆45Updated 2 months ago
• srush / raspy
  An interactive exploration of Transformer programming.
  ☆256Updated last year
• dvruette / barrel-rec-pytorch
  ☆53Updated last year
• keerthanpg / DadJokeGenerator
  ☆45Updated last year
• Aleph-Alpha / scaling
  Scaling is a distributed training library and installable dependency designed to scale up neural networks, with a dedicated module for tr…
  ☆53Updated 3 months ago
• justinchiu / openlogprobs
  Extract full next-token probabilities via language model APIs
  ☆229Updated 11 months ago
• warner-benjamin / commented-transformers
  Highly commented implementations of Transformers in PyTorch
  ☆132Updated last year
• KhoomeiK / complexity-scaling
  gzip Predicts Data-dependent Scaling Laws
  ☆33Updated 8 months ago
• irregular-rhomboid / EAI-Math-Reading-Group
  Resources from the EleutherAI Math Reading Group
  ☆52Updated last month
• huggingface / bloom-jax-inference
  ☆64Updated 2 years ago
• abacaj / train-with-fsdp
  ☆92Updated last year
• srush / Tensor-Puzzles-Penzai
  ☆20Updated 9 months ago
• srush / anynp
  Proof-of-concept of global switching between numpy/jax/pytorch in a library.
  ☆18Updated 7 months ago
• kernelmachine / cbtm
  Code repository for the c-BTM paper
  ☆105Updated last year