Systemcluster / kitokenLinks
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆40Updated 3 months ago
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below
Sorting:
- Locality Sensitive Hashing☆78Updated 2 years ago
- Inference engine for GLiNER models, in Rust☆82Updated last month
- A high-performance constrained decoding engine based on context free grammar in Rust☆57Updated 7 months ago
- Optimizing bit-level Jaccard Index and Population Counts for large-scale quantized Vector Search via Harley-Seal CSA and Lookup Tables☆21Updated 7 months ago
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆224Updated this week
- HSNW module for Redis☆59Updated 5 years ago
- Proof of concept for running moshi/hibiki using webrtc☆19Updated 10 months ago
- Rust crate for some audio utilities☆25Updated 10 months ago
- ☆39Updated 3 years ago
- A complete(grpc service and lib) Rust inference with multilingual embedding support. This version leverages the power of Rust for both GR…☆39Updated last year
- Modular Rust transformer/LLM library using Candle☆37Updated last year
- ☆82Updated 2 months ago
- Official Rust Implementation of Model2Vec☆145Updated 3 months ago
- Vector Database with support for late interaction and token level embeddings.☆54Updated 6 months ago
- Tree-based indexes for neural-search☆31Updated last year
- Code repository for the paper - "AdANNS: A Framework for Adaptive Semantic Search"☆66Updated 2 years ago
- Tooling for exact and MinHash deduplication of large-scale text datasets☆51Updated this week
- NLP with Rust for Python 🦀🐍☆70Updated 7 months ago
- Library for fast text representation and classification.☆31Updated 2 years ago
- Plug-and-play Search Interfaces with Pyserini and Hugging Face☆32Updated 2 years ago
- UniSim is a package for efficient similarity computation, fuzzy matching, and clustering of data.☆145Updated 9 months ago
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆67Updated 3 months ago
- A client library in Rust for Nvidia Triton.☆30Updated 2 years ago
- GGML implementation of BERT model with Python bindings and quantization.☆58Updated last year
- ☆12Updated last year
- High-Performance K-Means Clustering Library☆39Updated 6 months ago
- Pre-train Static Word Embeddings☆94Updated 4 months ago
- A file utility for accessing both local and remote files through a unified interface.☆45Updated 3 weeks ago
- Neural syntax annotator, supporting sequence labeling, lemmatization, and dependency parsing.☆80Updated 2 years ago
- Efficiently computing & storing token n-grams from large corpora☆26Updated last year