Systemcluster / kitokenLinks
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆29Updated 5 months ago
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below
Sorting:
- Inference engine for GLiNER models, in Rust☆64Updated last month
- A complete(grpc service and lib) Rust inference with multilingual embedding support. This version leverages the power of Rust for both GR…☆39Updated last year
- ☆39Updated 2 years ago
- Proof of concept for running moshi/hibiki using webrtc☆20Updated 5 months ago
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆199Updated last month
- HSNW module for Redis☆57Updated 4 years ago
- Tree-based indexes for neural-search☆32Updated last year
- Vector Database with support for late interaction and token level embeddings.☆55Updated 2 months ago
- Optimizing bit-level Jaccard Index and Population Counts for large-scale quantized Vector Search via Harley-Seal CSA and Lookup Tables☆20Updated 3 months ago
- A library for simplifying fine tuning with multi gpu setups in the Huggingface ecosystem.☆16Updated 9 months ago
- ☆76Updated 8 months ago
- This is the repo for the container that holds the models for the text2vec-transformers module☆54Updated 3 weeks ago
- A high-performance constrained decoding engine based on context free grammar in Rust☆55Updated 3 months ago
- GGML implementation of BERT model with Python bindings and quantization.☆57Updated last year
- Trace LLM calls (and others) and visualize them in WandB, as interactive SVG or using a streaming local webapp☆14Updated 6 months ago
- Sentence Embedding as a Service☆15Updated last month
- Simple high-throughput inference library☆127Updated 3 months ago
- Locality Sensitive Hashing☆72Updated 2 years ago
- XTR/WARP (SIGIR'25) is an extremely fast and accurate retrieval engine based on Stanford's ColBERTv2/PLAID and Google DeepMind's XTR.☆154Updated 3 months ago
- Pre-train Static Word Embeddings☆85Updated 2 months ago
- ☆12Updated 6 months ago
- Library for fast text representation and classification.☆31Updated last year
- wasm bindings for huggingface tokenizers library☆34Updated 3 years ago
- Rust crate for some audio utilities☆26Updated 5 months ago
- NLP with Rust for Python 🦀🐍☆64Updated 3 months ago
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆63Updated 2 weeks ago
- Plug-and-play Search Interfaces with Pyserini and Hugging Face☆32Updated 2 years ago
- implement llava using candle☆15Updated last year
- Code repository for the paper - "AdANNS: A Framework for Adaptive Semantic Search"☆65Updated last year
- ☆68Updated last month