NovelAI / novelai-tokenizer

Sentencepiece based BPE tokenizer for English and Japanese language text.
28Updated last year

Alternatives and similar repositories for novelai-tokenizer

Users that are interested in novelai-tokenizer are comparing it to the libraries listed below

Sorting: