sradc / pretraining-BERT
Pre-train BERT from scratch, with HuggingFace. Accompanies the blog post: sidsite.com/posts/bert-from-scratch
☆39Updated last year
Related projects ⓘ
Alternatives and complementary repositories for pretraining-BERT
- gzip Predicts Data-dependent Scaling Laws☆32Updated 5 months ago
- Jax like function transformation engine but micro, microjax☆26Updated 3 weeks ago
- ☆22Updated last year
- 📝 Reference-Free automatic summarization evaluation with potential hallucination detection☆98Updated 10 months ago
- Your favourite classical machine learning algos on the GPU/TPU☆20Updated 2 months ago
- ☆57Updated 11 months ago
- Scripts to prep PC for development use after OS installs☆37Updated last week
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆81Updated last year
- A lightweight PyTorch implementation of the Transformer-XL architecture proposed by Dai et al. (2019)☆35Updated last year
- ☆36Updated 3 months ago
- Our solution for the arc challenge 2024☆33Updated last week
- Evaluating the Mamba architecture on the Othello game☆43Updated 6 months ago
- ☆77Updated 7 months ago
- This is the code that went into our practical dive using mamba as information extraction☆50Updated 11 months ago
- ☆49Updated 8 months ago
- Simplified implementation of UMAP like dimensionality reduction algorithm☆40Updated this week
- ReLM is a Regular Expression engine for Language Models☆104Updated last year
- A MAD laboratory to improve AI architecture designs 🧪☆95Updated 6 months ago
- LLM training in simple, raw C/CUDA☆12Updated last month
- Swarming algorithms like PSO, Ant Colony, Sakana, and more in PyTorch 😊☆111Updated last week
- Code for NeurIPS LLM Efficiency Challenge☆54Updated 7 months ago
- Zeus LLM Trainer is a rewrite of Stanford Alpaca aiming to be the trainer for all Large Language Models☆69Updated last year
- ☆91Updated last year
- A library for squeakily cleaning and filtering language datasets.☆45Updated last year
- Serialize JAX, Flax, Haiku, or Objax model params with 🤗`safetensors`☆42Updated 5 months ago
- FastFeedForward Networks☆18Updated 11 months ago
- Functional local implementations of main model parallelism approaches☆95Updated last year
- An introduction to LLM Sampling☆64Updated 2 weeks ago
- Teaching Addition to Small Transformers☆14Updated 11 months ago
- A place to store reusable transformer components of my own creation or found on the interwebs☆44Updated 2 weeks ago