pchizhov / picky_bpe

BPE modification that implements removing of the intermediate tokens during tokenizer training.
25Updated last month

Alternatives and similar repositories for picky_bpe:

Users that are interested in picky_bpe are comparing it to the libraries listed below