sradc / pretraining-BERTLinks
Pre-train BERT from scratch, with HuggingFace. Accompanies the blog post: sidsite.com/posts/bert-from-scratch
☆40Updated 2 weeks ago
Alternatives and similar repositories for pretraining-BERT
Users that are interested in pretraining-BERT are comparing it to the libraries listed below
Sorting:
- gzip Predicts Data-dependent Scaling Laws☆35Updated last year
- ☆22Updated last year
- QLoRA for Masked Language Modeling☆22Updated last year
- Your favourite classical machine learning algos on the GPU/TPU☆20Updated 5 months ago
- ☆60Updated last year
- ☆54Updated 6 months ago
- ☆27Updated 10 months ago
- ☆53Updated last year
- ☆92Updated last year
- Genalog is an open source, cross-platform python package allowing generation of synthetic document images with custom degradations and te…☆42Updated last year
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆82Updated last year
- Code implementing "Efficient Parallelization of a Ubiquitious Sequential Computation" (Heinsen, 2023)☆94Updated 6 months ago
- A lightweight PyTorch implementation of the Transformer-XL architecture proposed by Dai et al. (2019)☆37Updated 2 years ago
- ☆49Updated last year
- JAX notebook showing how to LoRA + GPTQ arbitrary models☆9Updated last year
- PyTorch implementation for MRL☆18Updated last year
- Jax like function transformation engine but micro, microjax☆32Updated 7 months ago
- A case study of efficient training of large language models using commodity hardware.☆69Updated 2 years ago
- Exploring finetuning public checkpoints on filter 8K sequences on Pile☆115Updated 2 years ago
- Functional local implementations of main model parallelism approaches☆95Updated 2 years ago
- A library for squeakily cleaning and filtering language datasets.☆47Updated last year
- ☆48Updated last year
- A place to store reusable transformer components of my own creation or found on the interwebs☆56Updated 3 weeks ago
- Scaling is a distributed training library and installable dependency designed to scale up neural networks, with a dedicated module for tr…☆62Updated 7 months ago
- Simple GRPO scripts and configurations.☆58Updated 4 months ago
- ☆80Updated last year
- some common Huggingface transformers in maximal update parametrization (µP)☆80Updated 3 years ago
- Automatically take good care of your preemptible TPUs☆36Updated 2 years ago
- Collection of autoregressive model implementation☆85Updated last month
- ☆68Updated 9 months ago