lenML / tokenizers

a lightweight no-dependency fork from transformers.js (only tokenizers)
16Updated this week

Alternatives and similar repositories for tokenizers:

Users that are interested in tokenizers are comparing it to the libraries listed below