Systemcluster / kitokenLinks
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆37Updated last month
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below
Sorting:
- Inference engine for GLiNER models, in Rust☆76Updated last week
- A complete(grpc service and lib) Rust inference with multilingual embedding support. This version leverages the power of Rust for both GR…☆39Updated last year
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆214Updated last month
- A high-performance constrained decoding engine based on context free grammar in Rust☆55Updated 5 months ago
- Modular Rust transformer/LLM library using Candle☆36Updated last year
- Proof of concept for running moshi/hibiki using webrtc☆19Updated 8 months ago
- Locality Sensitive Hashing☆74Updated 2 years ago
- Rust crate for some audio utilities☆25Updated 8 months ago
- A client library in Rust for Nvidia Triton.☆30Updated 2 years ago
- HSNW module for Redis☆58Updated 5 years ago
- Vector Database with support for late interaction and token level embeddings.☆55Updated 4 months ago
- implement llava using candle☆15Updated last year
- Tree-based indexes for neural-search☆32Updated last year
- Trace LLM calls (and others) and visualize them in WandB, as interactive SVG or using a streaming local webapp☆14Updated 8 months ago
- A Demo server serving Bert through ONNX with GPU written in Rust with <3☆41Updated 4 years ago
- ☆79Updated last week
- Optimizing bit-level Jaccard Index and Population Counts for large-scale quantized Vector Search via Harley-Seal CSA and Lookup Tables☆21Updated 5 months ago
- GPU accelerated client-side embeddings for vector search, RAG etc.☆65Updated last year
- ☆39Updated 3 years ago
- NLP with Rust for Python 🦀🐍☆66Updated 6 months ago
- Faster Learned Sparse Retrieval with Block-Max Pruning. ACM SIGIR 2024.☆31Updated last month
- Pre-train Static Word Embeddings☆90Updated 2 months ago
- Run ONNX and TensorFlow inference in the browser.☆75Updated 2 years ago
- GGML implementation of BERT model with Python bindings and quantization.☆56Updated last year
- ☆12Updated last year
- Simple high-throughput inference library☆149Updated 6 months ago
- Efficient BM25 with DuckDB 🦆☆58Updated 10 months ago
- Sentence Embedding as a Service☆15Updated 4 months ago
- Code repository for the paper - "AdANNS: A Framework for Adaptive Semantic Search"☆65Updated 2 years ago
- Efficiently computing & storing token n-grams from large corpora☆26Updated last year