Systemcluster / kitokenLinks
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆24Updated 2 months ago
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below
Sorting:
- Proof of concept for running moshi/hibiki using webrtc☆19Updated 3 months ago
- Modular Rust transformer/LLM library using Candle☆35Updated last year
- A high-performance constrained decoding engine based on context free grammar in Rust☆52Updated 2 weeks ago
- Efficient BM25 with DuckDB 🦆☆49Updated 5 months ago
- Rust bindings for CTranslate2☆14Updated last year
- Locality Sensitive Hashing☆71Updated last year
- ☆39Updated 2 years ago
- ☆11Updated 4 months ago
- Tree-based indexes for neural-search☆32Updated last year
- Optimizing bit-level Jaccard Index and Population Counts for large-scale quantized Vector Search via Harley-Seal CSA and Lookup Tables☆20Updated 2 weeks ago
- A complete(grpc service and lib) Rust inference with multilingual embedding support. This version leverages the power of Rust for both GR…☆39Updated 9 months ago
- GGML implementation of BERT model with Python bindings and quantization.☆54Updated last year
- A small rust-based data loader☆24Updated 5 months ago
- Vector Database with support for late interaction and token level embeddings.☆54Updated 8 months ago
- Model implementation for the contextual embeddings project☆26Updated this week
- Showcase how mxbai-embed-large-v1 can be used to produce binary embedding. Binary embeddings enabled 32x storage savings and 40x faster r…☆18Updated last year
- HSNW module for Redis☆57Updated 4 years ago
- Rust wrapper for Microsoft's ONNX Runtime with CUDA support (version 1.7)☆23Updated 2 years ago
- A library for simplifying fine tuning with multi gpu setups in the Huggingface ecosystem.☆16Updated 7 months ago
- Library for fast text representation and classification.☆28Updated last year
- j1-micro (1.7B) & j1-nano (600M) are absurdly tiny but mighty reward models.☆74Updated last week
- Rust crate for some audio utilities☆23Updated 2 months ago
- A Demo server serving Bert through ONNX with GPU written in Rust with <3☆40Updated 3 years ago
- Inference engine for GLiNER models, in Rust☆58Updated 2 months ago
- implement llava using candle☆15Updated 11 months ago
- A file utility for accessing both local and remote files through a unified interface.☆42Updated 3 weeks ago
- ANE accelerated embedding models!☆17Updated 5 months ago
- Pre-train Static Word Embeddings☆70Updated this week
- Latent Large Language Models☆18Updated 9 months ago
- This demo showcase the use of onnxruntime-rs with a GPU on CUDA 11 to run Bert in a data pipeline with Rust.☆16Updated 3 years ago