Systemcluster / kitokenLinks
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆26Updated 3 months ago
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below
Sorting:
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆187Updated 3 weeks ago
- A high-performance constrained decoding engine based on context free grammar in Rust☆54Updated last month
- Optimizing bit-level Jaccard Index and Population Counts for large-scale quantized Vector Search via Harley-Seal CSA and Lookup Tables☆20Updated last month
- Locality Sensitive Hashing☆72Updated 2 years ago
- ☆39Updated 2 years ago
- Inference engine for GLiNER models, in Rust☆61Updated last week
- GGML implementation of BERT model with Python bindings and quantization.☆55Updated last year
- ☆11Updated 5 months ago
- implement llava using candle☆15Updated last year
- A complete(grpc service and lib) Rust inference with multilingual embedding support. This version leverages the power of Rust for both GR…☆39Updated 10 months ago
- Rust bindings for CTranslate2☆14Updated 2 years ago
- Pre-train Static Word Embeddings☆84Updated last month
- Vector Database with support for late interaction and token level embeddings.☆55Updated 3 weeks ago
- Implementation of the paper "Lossless Compression of Vector IDs for Approximate Nearest Neighbor Search" by Severo et al.☆80Updated 5 months ago
- Library for fast text representation and classification.☆30Updated last year
- NLP with Rust for Python 🦀🐍☆63Updated 2 months ago
- Rust crate for some audio utilities☆26Updated 4 months ago
- Code repository for the paper - "AdANNS: A Framework for Adaptive Semantic Search"☆65Updated last year
- Improving Text Embedding of Language Models Using Contrastive Fine-tuning☆64Updated 11 months ago
- Truly flash T5 realization!☆68Updated last year
- Showcase how mxbai-embed-large-v1 can be used to produce binary embedding. Binary embeddings enabled 32x storage savings and 40x faster r…☆18Updated last year
- Cortex-compatible model server for Python and TensorFlow☆17Updated 2 years ago
- Tree-based indexes for neural-search☆32Updated last year
- Proof of concept for running moshi/hibiki using webrtc☆20Updated 4 months ago
- XTR/WARP (SIGIR'25) is an extremely fast and accurate retrieval engine based on Stanford's ColBERTv2/PLAID and Google DeepMind's XTR.☆137Updated 2 months ago
- ANE accelerated embedding models!☆18Updated 7 months ago
- HSNW module for Redis☆57Updated 4 years ago
- ☆62Updated last week
- Python library to use Pleias-RAG models☆58Updated 2 months ago
- Plug-and-play Search Interfaces with Pyserini and Hugging Face☆32Updated last year