kuprel / minbpe-pytorch

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization, with PyTorch/CUDA
35Updated 11 months ago

Alternatives and similar repositories for minbpe-pytorch:

Users that are interested in minbpe-pytorch are comparing it to the libraries listed below