transitive-bullshit / compare-tokenizers

A test suite comparing Node.js BPE tokenizers for use with AI models.
29Updated 11 months ago

Related projects

Alternatives and complementary repositories for compare-tokenizers