guillaume-be / rust-tokenizersLinks
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
☆319Updated last year
Alternatives and similar repositories for rust-tokenizers
Users that are interested in rust-tokenizers are comparing it to the libraries listed below
Sorting:
- Rust implementation of the HNSW algorithm (Malkov-Yashunin)☆192Updated last week
- Rust port of sentence-transformers (https://github.com/UKPLab/sentence-transformers)☆115Updated 9 months ago
- Rust wrapper for Microsoft's ONNX Runtime (version 1.8)☆297Updated last year
- Inference Llama 2 in one file of pure Rust 🦀☆233Updated last year
- Rust language bindings for Faiss