Systemcluster / kitoken
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆19Updated last week
Alternatives and similar repositories for kitoken:
Users that are interested in kitoken are comparing it to the libraries listed below
- Library for fast text representation and classification.☆28Updated last year
- Binary vector search example using Unum's USearch engine and pre-computed Wikipedia embeddings from Co:here and MixedBread☆18Updated 11 months ago
- utilities for loading and running text embeddings with onnx☆44Updated 7 months ago
- A library for squeakily cleaning and filtering language datasets.☆46Updated last year
- ☆37Updated 2 years ago
- Latent Large Language Models☆17Updated 7 months ago
- NLP with Rust for Python 🦀🐍☆61Updated 9 months ago
- Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https:/…☆26Updated 11 months ago
- Versatile framework designed to streamline the integration of your models, as well as those sourced from Hugging Face, into complex progr…☆28Updated 3 months ago
- Rust bindings for CTranslate2☆14Updated last year
- Pre-train Static Word Embeddings☆50Updated 3 weeks ago
- ANE accelerated embedding models!☆17Updated 3 months ago
- Tokenization across languages. Useful as preprocessing for subword tokenization.☆22Updated 2 years ago
- Trace LLM calls (and others) and visualize them in WandB, as interactive SVG or using a streaming local webapp☆14Updated last month
- Creating Generative AI Apps which work☆17Updated 8 months ago
- Showcase how mxbai-embed-large-v1 can be used to produce binary embedding. Binary embeddings enabled 32x storage savings and 40x faster r…☆15Updated last year
- Plug-and-play Search Interfaces with Pyserini and Hugging Face☆31Updated last year
- Python package for generating datasets to evaluate reasoning and retrieval of large language models☆15Updated this week
- Using modal.com to process FineWeb-edu data☆20Updated 3 weeks ago
- Efficient BM25 with DuckDB 🦆☆44Updated 3 months ago
- a pipeline for using api calls to agnostically convert unstructured data into structured training data☆30Updated 6 months ago
- Starbucks: Improved Training for 2D Matryoshka Embeddings☆18Updated last month
- Vector Database with support for late interaction and token level embeddings.☆53Updated 5 months ago
- Tree-based indexes for neural-search☆29Updated last year
- QLoRA for Masked Language Modeling☆21Updated last year
- ☆38Updated last month
- ☆8Updated 8 months ago
- Cortex-compatible model server for Python and TensorFlow☆17Updated 2 years ago
- Genalog is an open source, cross-platform python package allowing generation of synthetic document images with custom degradations and te…☆42Updated last year
- ☆21Updated 3 weeks ago