kuprel / minbpe-pytorch
View external linksLinks

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization, with PyTorch/CUDA
42Feb 27, 2024Updated last year

Alternatives and similar repositories for minbpe-pytorch

Users that are interested in minbpe-pytorch are comparing it to the libraries listed below

Sorting:

Are these results useful?