Alternatives and similar repositories for pretraining-BERT:

Users that are interested in pretraining-BERT are comparing it to the libraries listed below

• NousResearch / StripedHyenaTrainer
  ☆60Updated last year
• KhoomeiK / complexity-scaling
  gzip Predicts Data-dependent Scaling Laws
  ☆34Updated 8 months ago
• joey00072 / microjax
  Jax like function transformation engine but micro, microjax
  ☆30Updated 3 months ago
• CarperAI / treasure_trove
  ☆22Updated last year
• drisspg / transformer_nuggets
  A place to store reusable transformer components of my own creation or found on the interwebs
  ☆46Updated this week
• augustwester / transformer-xl
  A lightweight PyTorch implementation of the Transformer-XL architecture proposed by Dai et al. (2019)
  ☆37Updated 2 years ago
• sytelus / pcprep
  Scripts to prep PC for development use after OS installs
  ☆37Updated last week
• ChrisHayduk / QLoRA-for-MLM
  QLoRA for Masked Language Modeling
  ☆21Updated last year
• epfml / DenseFormer
  ☆78Updated 10 months ago
• rasbt / nn_plus_gzip
  Gzip and nearest neighbors for text classification
  ☆56Updated last year
• LiibanMo / scikit-jax
  Your favourite classical machine learning algos on the GPU/TPU
  ☆20Updated last month
• Pleias / Quest-Best-Tokens
  An introduction to LLM Sampling
  ☆75Updated 2 months ago
• jfpuget / ARC-AGI-Challenge-2024
  ☆47Updated 2 months ago
• pchizhov / picky_bpe
  BPE modification that implements removing of the intermediate tokens during tokenizer training.
  ☆25Updated 2 months ago
• abacaj / train-with-fsdp
  ☆92Updated last year
• krypticmouse / matryoshka-representation-learning
  PyTorch implementation for MRL
  ☆18Updated 11 months ago
• Alignment-Lab-AI / datagen
  a pipeline for using api calls to agnostically convert unstructured data into structured training data
  ☆29Updated 4 months ago
• taylorai / onnx_embedding_models
  utilities for loading and running text embeddings with onnx
  ☆44Updated 6 months ago
• lucidrains / gateloop-transformer
  Implementation of GateLoop Transformer in Pytorch and Jax
  ☆87Updated 8 months ago
• unicamp-dl / InRanker
  ☆46Updated last year
• johnma2006 / candle
  Deep learning library implemented from scratch in numpy. Mixtral, Mamba, LLaMA, GPT, ResNet, and other experiments.
  ☆51Updated 10 months ago
• joey00072 / ohara
  Collection of autoregressive model implementation
  ☆81Updated last week
• dvruette / barrel-rec-pytorch
  ☆53Updated last year
• RobertCsordas / moe
  Official repository for the paper "Approximating Two-Layer Feedforward Networks for Efficient Transformers"
  ☆36Updated last year
• euclaise / supertrainer2000
  ☆49Updated 11 months ago
• AblateIt / finetune-study
  Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.
  ☆82Updated last year
• amirzandieh / HyperAttention
  Triton Implementation of HyperAttention Algorithm
  ☆46Updated last year
• luyug / magix
  Supercharge huggingface transformers with model parallelism.
  ☆76Updated 4 months ago
• Upaya07 / NeurIPS-llm-efficiency-challenge
  Code for NeurIPS LLM Efficiency Challenge
  ☆55Updated 10 months ago