Alternatives and similar repositories for pretraining-BERT:

Users that are interested in pretraining-BERT are comparing it to the libraries listed below

• KhoomeiK / complexity-scaling
  gzip Predicts Data-dependent Scaling Laws
  ☆34Updated 10 months ago
• drisspg / transformer_nuggets
  A place to store reusable transformer components of my own creation or found on the interwebs
  ☆48Updated last week
• ChrisHayduk / qlora-multi-gpu
  QLoRA with Enhanced Multi GPU Support
  ☆36Updated last year
• NousResearch / StripedHyenaTrainer
  ☆60Updated last year
• CarperAI / treasure_trove
  ☆22Updated last year
• LiibanMo / scikit-jax
  Your favourite classical machine learning algos on the GPU/TPU
  ☆20Updated 2 months ago
• stas00 / ml-ways
  ML/DL Math and Method notes
  ☆59Updated last year
• krypticmouse / matryoshka-representation-learning
  PyTorch implementation for MRL
  ☆18Updated last year
• HomebrewML / HomebrewNLP-torch
  A case study of efficient training of large language models using commodity hardware.
  ☆69Updated 2 years ago
• abacaj / train-with-fsdp
  ☆92Updated last year
• srush / Tensor-Puzzles-Penzai
  ☆20Updated 11 months ago
• rwightman / genalog
  Genalog is an open source, cross-platform python package allowing generation of synthetic document images with custom degradations and te…
  ☆42Updated last year
• domluna / pubmedFastRAG
  ☆28Updated last month
• lucidrains / gateloop-transformer
  Implementation of GateLoop Transformer in Pytorch and Jax
  ☆87Updated 9 months ago
• joey00072 / microjax
  Jax like function transformation engine but micro, microjax
  ☆30Updated 5 months ago
• lucidrains / genetic-algorithm-pytorch
  Toy genetic algorithm in Pytorch
  ☆33Updated this week
• joey00072 / ohara
  Collection of autoregressive model implementation
  ☆83Updated last month
• rasbt / pytorch-memory-optim
  This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…
  ☆87Updated last year
• augustwester / transformer-xl
  A lightweight PyTorch implementation of the Transformer-XL architecture proposed by Dai et al. (2019)
  ☆37Updated 2 years ago
• luyug / magix
  Supercharge huggingface transformers with model parallelism.
  ☆76Updated 5 months ago
• Pleias / Quest-Best-Tokens
  An introduction to LLM Sampling
  ☆77Updated 3 months ago
• jfpuget / ARC-AGI-Challenge-2024
  ☆47Updated 4 months ago
• lessw2020 / transformer_central
  Various transformers for FSDP research
  ☆37Updated 2 years ago
• lucidrains / grokfast-pytorch
  Explorations into the proposal from the paper "Grokfast, Accelerated Grokking by Amplifying Slow Gradients"
  ☆98Updated 3 months ago
• epfml / DenseFormer
  ☆79Updated 11 months ago
• kmkolasinski / nano-umap
  Simplified implementation of UMAP like dimensionality reduction algorithm
  ☆48Updated 4 months ago
• hundredblocks / large-model-parallelism
  Functional local implementations of main model parallelism approaches
  ☆95Updated 2 years ago
• ElleLeonne / Lightning-ReLoRA
  A public implementation of the ReLoRA pretraining method, built on Lightning-AI's Pytorch Lightning suite.
  ☆33Updated last year
• athms / mad-lab
  A MAD laboratory to improve AI architecture designs 🧪
  ☆108Updated 3 months ago
• okarthikb / state-space-models
  ☆27Updated 8 months ago