Alternatives and similar repositories for pretraining-BERT

Users that are interested in pretraining-BERT are comparing it to the libraries listed below

• abacaj / train-with-fsdp
  ☆94Updated 2 years ago
• hundredblocks / large-model-parallelism
  Functional local implementations of main model parallelism approaches
  ☆95Updated 2 years ago
• KhoomeiK / complexity-scaling
  gzip Predicts Data-dependent Scaling Laws
  ☆34Updated last year
• rasbt / pytorch-memory-optim
  This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…
  ☆92Updated 2 years ago
• NousResearch / StripedHyenaTrainer
  ☆62Updated 2 years ago
• muellerzr / minimal-trainer-zoo
  Minimal example scripts of the Hugging Face Trainer, focused on staying under 150 lines
  ☆196Updated last year
• luyug / magix
  Supercharge huggingface transformers with model parallelism.
  ☆78Updated 6 months ago
• eduardoleao052 / Autograd-from-scratch
  Documented and Unit Tested educational Deep Learning framework with Autograd from scratch.
  ☆122Updated last year
• Pleias / Quest-Best-Tokens
  An introduction to LLM Sampling
  ☆79Updated last year
• Muhtasham / summarization-eval
  📝 Reference-Free automatic summarization evaluation with potential hallucination detection
  ☆103Updated 2 years ago
• mkuchnik / relm
  ReLM is a Regular Expression engine for Language Models
  ☆107Updated 2 years ago
• rwightman / genalog
  Genalog is an open source, cross-platform python package allowing generation of synthetic document images with custom degradations and te…
  ☆44Updated 2 years ago
• rasbt / nn_plus_gzip
  Gzip and nearest neighbors for text classification
  ☆57Updated 2 years ago
• warner-benjamin / commented-transformers
  Highly commented implementations of Transformers in PyTorch
  ☆138Updated 2 years ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆103Updated last year
• RobertRiachi / nanoPALM
  ☆144Updated 2 years ago
• CarperAI / treasure_trove
  ☆22Updated 2 years ago
• HomebrewML / HomebrewNLP-torch
  A case study of efficient training of large language models using commodity hardware.
  ☆68Updated 3 years ago
• sytelus / pcprep
  Various handy scripts to quickly setup new Linux and Windows sandboxes, containers and WSL.
  ☆40Updated this week
• muellerzr / nbdistributed
  Seemless interface of using PyTOrch distributed with Jupyter notebooks
  ☆57Updated 4 months ago
• ESHyperscale / nano-egg
  Evolution Pretraining Fully in Int Formats
  ☆136Updated 2 months ago
• jxiw / BiGS
  Official Repository of Pretraining Without Attention (BiGS), BiGS is the first model to achieve BERT-level transfer learning on the GLUE …
  ☆116Updated last year
• drisspg / transformer_nuggets
  A place to store reusable transformer components of my own creation or found on the interwebs
  ☆72Updated this week
• joey00072 / microjax
  Jax like function transformation engine but micro, microjax
  ☆34Updated last year
• jxbz / agd
  Automatic gradient descent
  ☆217Updated 2 years ago
• davisyoshida / easy-lora-and-gptq
  JAX notebook showing how to LoRA + GPTQ arbitrary models
  ☆10Updated 2 years ago
• jfpuget / ARC-AGI-Challenge-2024
  ☆56Updated last year
• KaiNylund / lm-weights-encode-time
  ☆68Updated last year
• krypticmouse / matryoshka-representation-learning
  PyTorch implementation for MRL
  ☆21Updated last year
• SumanthRH / tokenization
  A comprehensive deep dive into the world of tokens
  ☆227Updated last year