pytorch-labs / tokenizers

C++ implementations for various tokenizers (sentencepiece, tiktoken etc).
19Updated last week

Alternatives and similar repositories for tokenizers:

Users that are interested in tokenizers are comparing it to the libraries listed below