Kshao09 / Pruned-BPE-TokenizerView on GitHub
A Python/Cython implementation of Pruned BPE, a Byte Pair Encoding tokenizer that improves visible vocabulary efficiency by pruning low-exposure tokens while preserving internal merge structure.
22Jul 24, 2026Updated this week

Alternatives and similar repositories for Pruned-BPE-Tokenizer

Users that are interested in Pruned-BPE-Tokenizer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

Sorting:

Are these results useful?