smallhelm / tokenizer2

tokenize any text stream given some basic regex rules to match tokens
14Updated 6 years ago

Alternatives and similar repositories for tokenizer2:

Users that are interested in tokenizer2 are comparing it to the libraries listed below