OpenLMLab / ParallelTokenizer

Use the tokenizer in parallel to achieve superior acceleration
16Updated last year

Alternatives and similar repositories for ParallelTokenizer:

Users that are interested in ParallelTokenizer are comparing it to the libraries listed below