NovelAI / novelai-tokenizer

Sentencepiece based BPE tokenizer for English and Japanese language text.
27Updated 9 months ago

Alternatives and similar repositories for novelai-tokenizer:

Users that are interested in novelai-tokenizer are comparing it to the libraries listed below