sradc / pretraining-BERT
Pre-train BERT from scratch, with HuggingFace. Accompanies the blog post: sidsite.com/posts/bert-from-scratch
☆37Updated last year
Related projects: ⓘ
- A lightweight PyTorch implementation of the Transformer-XL architecture proposed by Dai et al. (2019)☆34Updated last year
- ☆55Updated 9 months ago
- gzip Predicts Data-dependent Scaling Laws☆31Updated 3 months ago
- A place to store reusable transformer components of my own creation or found on the interwebs☆43Updated 3 weeks ago
- Jax like function transformation engine but micro, microjax☆24Updated 3 weeks ago
- ☆22Updated last year
- Collection of autoregressive model implementation☆62Updated 2 weeks ago
- ☆73Updated 5 months ago
- ☆89Updated 11 months ago
- Deep learning library implemented from scratch in numpy. Mixtral, Mamba, LLaMA, GPT, ResNet, and other experiments.☆47Updated 5 months ago
- ☆53Updated 8 months ago
- Your favourite classical machine learning algos on the GPU/TPU☆20Updated this week
- A MAD laboratory to improve AI architecture designs 🧪☆84Updated 4 months ago
- Serialize JAX, Flax, Haiku, or Objax model params with 🤗`safetensors`☆41Updated 3 months ago
- Implementation of GateLoop Transformer in Pytorch and Jax☆86Updated 3 months ago
- a pipeline for using api calls to agnostically convert unstructured data into structured training data☆26Updated last year
- Code implementing "Efficient Parallelization of a Ubiquitious Sequential Computation" (Heinsen, 2023)☆74Updated 8 months ago
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆81Updated last year
- Clean RL implementation using MLX☆26Updated 6 months ago
- ☆27Updated 2 months ago
- ☆48Updated 6 months ago
- Sparse and discrete interpretability tool for neural networks☆51Updated 7 months ago
- ☆42Updated 3 months ago
- This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…☆84Updated last year
- My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other h…☆51Updated last year
- Full finetuning of large language models without large memory requirements☆94Updated 8 months ago
- Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount…☆50Updated 10 months ago
- A library for squeakily cleaning and filtering language datasets.☆45Updated last year
- ☆68Updated 2 months ago
- ☆48Updated 11 months ago
- ☆17Updated 4 months ago