sradc / pretraining-BERT
Pre-train BERT from scratch, with HuggingFace. Accompanies the blog post: sidsite.com/posts/bert-from-scratch
☆40Updated last year
Alternatives and similar repositories for pretraining-BERT
Users that are interested in pretraining-BERT are comparing it to the libraries listed below
Sorting:
- gzip Predicts Data-dependent Scaling Laws☆35Updated 11 months ago
- ☆92Updated last year
- ☆61Updated last year
- Your favourite classical machine learning algos on the GPU/TPU☆20Updated 4 months ago
- ☆22Updated last year
- Simplified implementation of UMAP like dimensionality reduction algorithm☆48Updated 5 months ago
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆82Updated last year
- ☆53Updated 5 months ago
- 📝 Reference-Free automatic summarization evaluation with potential hallucination detection☆100Updated last year
- Functional local implementations of main model parallelism approaches☆95Updated 2 years ago
- My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other h…☆53Updated last year
- Gzip and nearest neighbors for text classification☆57Updated last year
- ML/DL Math and Method notes☆60Updated last year
- PyTorch implementation for MRL☆18Updated last year
- QLoRA for Masked Language Modeling☆22Updated last year
- Scaling is a distributed training library and installable dependency designed to scale up neural networks, with a dedicated module for tr…☆60Updated 6 months ago
- a pipeline for using api calls to agnostically convert unstructured data into structured training data☆30Updated 7 months ago
- A place to store reusable transformer components of my own creation or found on the interwebs☆55Updated this week
- Collection of autoregressive model implementation☆85Updated 3 weeks ago
- ☆40Updated last year
- Full finetuning of large language models without large memory requirements☆94Updated last year
- Genalog is an open source, cross-platform python package allowing generation of synthetic document images with custom degradations and te…☆42Updated last year
- LLM training in simple, raw C/CUDA☆14Updated 5 months ago
- Deep learning library implemented from scratch in numpy. Mixtral, Mamba, LLaMA, GPT, ResNet, and other experiments.☆51Updated last year
- Optimizing bit-level Jaccard Index and Population Counts for large-scale quantized Vector Search via Harley-Seal CSA and Lookup Tables☆18Updated this week
- ☆55Updated 6 months ago
- ☆20Updated last year
- Inference code for LLaMA models in JAX☆118Updated 11 months ago
- some common Huggingface transformers in maximal update parametrization (µP)☆80Updated 3 years ago
- A case study of efficient training of large language models using commodity hardware.☆69Updated 2 years ago