transitive-bullshit / compare-tokenizers

A test suite comparing Node.js BPE tokenizers for use with AI models.
29Updated last year

Alternatives and similar repositories for compare-tokenizers:

Users that are interested in compare-tokenizers are comparing it to the libraries listed below