NovelAI / novelai-tokenizer

Sentencepiece based BPE tokenizer for English and Japanese language text.
26Updated 7 months ago

Related projects

Alternatives and complementary repositories for novelai-tokenizer