Alternatives and similar repositories for pretraining-BERT:

Users that are interested in pretraining-BERT are comparing it to the libraries listed below

• KhoomeiK / complexity-scaling
  gzip Predicts Data-dependent Scaling Laws
  ☆33Updated 7 months ago
• NousResearch / StripedHyenaTrainer
  ☆60Updated last year
• drisspg / transformer_nuggets
  A place to store reusable transformer components of my own creation or found on the interwebs
  ☆44Updated this week
• augustwester / transformer-xl
  A lightweight PyTorch implementation of the Transformer-XL architecture proposed by Dai et al. (2019)
  ☆37Updated last year
• hundredblocks / large-model-parallelism
  Functional local implementations of main model parallelism approaches
  ☆95Updated last year
• microsoft / mutransformers
  some common Huggingface transformers in maximal update parametrization (µP)
  ☆79Updated 2 years ago
• epfml / DenseFormer
  ☆78Updated 9 months ago
• jfpuget / ARC-AGI-Challenge-2024
  ☆46Updated last month
• AblateIt / finetune-study
  Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.
  ☆82Updated last year
• kmkolasinski / nano-umap
  Simplified implementation of UMAP like dimensionality reduction algorithm
  ☆44Updated 2 months ago
• okarthikb / state-space-models
  ☆27Updated 6 months ago
• LiibanMo / scikit-jax
  Your favourite classical machine learning algos on the GPU/TPU
  ☆20Updated 2 weeks ago
• joey00072 / microjax
  Jax like function transformation engine but micro, microjax
  ☆30Updated 2 months ago
• athms / mad-lab
  A MAD laboratory to improve AI architecture designs 🧪
  ☆102Updated last month
• Aleph-Alpha / scaling
  Scaling is a distributed training library and installable dependency designed to scale up neural networks, with a dedicated module for tr…
  ☆53Updated 2 months ago
• krypticmouse / matryoshka-representation-learning
  PyTorch implementation for MRL
  ☆18Updated 10 months ago
• irregular-rhomboid / EAI-Math-Reading-Group
  Resources from the EleutherAI Math Reading Group
  ☆52Updated last month
• HazyResearch / train-tk
  train with kittens!
  ☆52Updated 2 months ago
• abacaj / train-with-fsdp
  ☆91Updated last year
• luyug / magix
  Supercharge huggingface transformers with model parallelism.
  ☆76Updated 3 months ago
• kyleliang919 / Long-context-transformers
  Exploring finetuning public checkpoints on filter 8K sequences on Pile
  ☆115Updated last year
• CarperAI / treasure_trove
  ☆22Updated last year
• HomebrewNLP / HomebrewNLP
  A case study of efficient training of large language models using commodity hardware.
  ☆68Updated 2 years ago
• domluna / pubmedFastRAG
  ☆28Updated 6 months ago
• Muhtasham / summarization-eval
  📝 Reference-Free automatic summarization evaluation with potential hallucination detection
  ☆99Updated last year
• alvarobartt / safejax
  Serialize JAX, Flax, Haiku, or Objax model params with 🤗`safetensors`
  ☆42Updated 7 months ago
• lucidrains / flash-genomics-model
  My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other h…
  ☆52Updated last year
• expz / annotated-hyena
  An annotated implementation of the Hyena Hierarchy paper
  ☆32Updated last year
• CarperAI / squeakily
  A library for squeakily cleaning and filtering language datasets.
  ☆45Updated last year
• joey00072 / ohara
  Collection of autoregressive model implementation
  ☆76Updated last week