Alternatives and similar repositories for pretraining-BERT

Users that are interested in pretraining-BERT are comparing it to the libraries listed below

• abacaj / train-with-fsdp
  ☆93Updated last year
• hundredblocks / large-model-parallelism
  Functional local implementations of main model parallelism approaches
  ☆96Updated 2 years ago
• eduardoleao052 / Autograd-from-scratch
  Documented and Unit Tested educational Deep Learning framework with Autograd from scratch.
  ☆120Updated last year
• warner-benjamin / commented-transformers
  Highly commented implementations of Transformers in PyTorch
  ☆137Updated 2 years ago
• rasbt / pytorch-memory-optim
  This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…
  ☆92Updated 2 years ago
• rasbt / nn_plus_gzip
  Gzip and nearest neighbors for text classification
  ☆57Updated 2 years ago
• muellerzr / minimal-trainer-zoo
  Minimal example scripts of the Hugging Face Trainer, focused on staying under 150 lines
  ☆197Updated last year
• KhoomeiK / complexity-scaling
  gzip Predicts Data-dependent Scaling Laws
  ☆35Updated last year
• HomebrewML / HomebrewNLP-torch
  A case study of efficient training of large language models using commodity hardware.
  ☆68Updated 3 years ago
• srush / raspy
  An interactive exploration of Transformer programming.
  ☆269Updated last year
• SumanthRH / tokenization
  A comprehensive deep dive into the world of tokens
  ☆226Updated last year
• sangmichaelxie / cs324_p2
  Project 2 (Building Large Language Models) for Stanford CS324: Understanding and Developing Large Language Models (Winter 2022)
  ☆105Updated 2 years ago
• stas00 / ml-ways
  ML/DL Math and Method notes
  ☆63Updated last year
• NousResearch / StripedHyenaTrainer
  ☆61Updated last year
• sytelus / pcprep
  Various handy scripts to quickly setup new Linux and Windows sandboxes, containers and WSL.
  ☆40Updated this week
• epfml / DenseFormer
  ☆82Updated last year
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated last year
• robertvacareanu / llm4regression
  Examining how large language models (LLMs) perform across various synthetic regression tasks when given (input, output) examples in their…
  ☆153Updated 11 months ago
• drisspg / transformer_nuggets
  A place to store reusable transformer components of my own creation or found on the interwebs
  ☆60Updated this week
• rwightman / genalog
  Genalog is an open source, cross-platform python package allowing generation of synthetic document images with custom degradations and te…
  ☆42Updated last year
• augustwester / transformer-xl
  A lightweight PyTorch implementation of the Transformer-XL architecture proposed by Dai et al. (2019)
  ☆37Updated 2 years ago
• glassroom / heinsen_sequence
  Code implementing "Efficient Parallelization of a Ubiquitious Sequential Computation" (Heinsen, 2023)
  ☆94Updated 8 months ago
• ConiferLabsWA / flan-ul2-dolly
  ☆34Updated 2 years ago
• jxmorris12 / bm25_pt
  minimal pytorch implementation of bm25 (with sparse tensors)
  ☆104Updated last year
• kmkolasinski / nano-umap
  Simplified implementation of UMAP like dimensionality reduction algorithm
  ☆50Updated 9 months ago
• RobertRiachi / nanoPALM
  ☆144Updated 2 years ago
• jxbz / agd
  Automatic gradient descent
  ☆210Updated 2 years ago
• ChrisHayduk / QLoRA-for-MLM
  QLoRA for Masked Language Modeling
  ☆22Updated last year
• lucidrains / llama-qrlhf
  Implementation of the Llama architecture with RLHF + Q-learning
  ☆166Updated 7 months ago
• HenryNdubuaku / nanodl
  A Jax-based library for building transformers, includes implementations of GPT, Gemma, LlaMa, Mixtral, Whisper, SWin, ViT and more.
  ☆290Updated last year