Systemcluster / kitokenLinks
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆26Updated 3 months ago
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below
Sorting:
- A high-performance constrained decoding engine based on context free grammar in Rust☆53Updated last month
- Proof of concept for running moshi/hibiki using webrtc☆19Updated 3 months ago
- Inference engine for GLiNER models, in Rust☆59Updated 2 months ago
- Rust crate for some audio utilities☆24Updated 3 months ago
- Showcase how mxbai-embed-large-v1 can be used to produce binary embedding. Binary embeddings enabled 32x storage savings and 40x faster r…☆18Updated last year
- Library for fast text representation and classification.☆30Updated last year
- Locality Sensitive Hashing☆72Updated last year
- Creating Generative AI Apps which work☆17Updated 2 months ago
- Efficient BM25 with DuckDB 🦆☆49Updated 6 months ago
- Vector Database with support for late interaction and token level embeddings.☆55Updated 8 months ago
- ☆11Updated 4 months ago
- NLP with Rust for Python 🦀🐍☆62Updated last month
- Optimizing bit-level Jaccard Index and Population Counts for large-scale quantized Vector Search via Harley-Seal CSA and Lookup Tables☆20Updated last month
- Pre-train Static Word Embeddings☆79Updated 3 weeks ago
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆58Updated last month
- ANE accelerated embedding models!☆18Updated 6 months ago
- Rust bindings for CTranslate2☆14Updated 2 years ago
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆177Updated this week
- A complete(grpc service and lib) Rust inference with multilingual embedding support. This version leverages the power of Rust for both GR…☆39Updated 10 months ago
- ☆72Updated 6 months ago
- implement llava using candle☆15Updated last year
- ☆47Updated 4 months ago
- ☆39Updated 2 years ago
- Simple high-throughput inference library☆119Updated last month
- A library for simplifying fine tuning with multi gpu setups in the Huggingface ecosystem.☆16Updated 7 months ago
- GGML implementation of BERT model with Python bindings and quantization.☆55Updated last year
- XTR/WARP (SIGIR'25) is an extremely fast and accurate retrieval engine based on Stanford's ColBERTv2/PLAID and Google DeepMind's XTR.☆131Updated last month
- Modular Rust transformer/LLM library using Candle☆36Updated last year
- WIP: Ofen is a toolkit aimed at making transformer models production-ready. API included☆16Updated 8 months ago
- A small python library to run iterators in a separate process☆10Updated last year