guillaume-be / rust-tokenizersLinks

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
315Updated last year

Alternatives and similar repositories for rust-tokenizers

Users that are interested in rust-tokenizers are comparing it to the libraries listed below

Sorting: