guillaume-be / rust-tokenizersLinks

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
326Updated 2 years ago

Alternatives and similar repositories for rust-tokenizers

Users that are interested in rust-tokenizers are comparing it to the libraries listed below

Sorting: