Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆62Oct 26, 2026Updated last week
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Conditional Random Fields implemented in pure Rust☆13Mar 17, 2025Updated last year
- Fast Tokens☆155Updated this week
- Use Jev (TypeSafe's System One model) as a calibrated reranker: one call, up to 30 documents, a probability per document. Apache-2.0.☆18Sep 17, 2026Updated last month
- Run all the tests at the same time with modal.com☆11Mar 2, 2024Updated 2 years ago
- Build script crate to run npm commands.☆20Nov 16, 2022Updated 3 years ago
- Open source password manager - Proton Pass • AdSecurely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
- A Rust crate offering similar functionality to the Python transformers package using Candle.☆15Nov 19, 2024Updated last year
- Sparton: Fast and Memory-Efficient Triton Kernel for Learned Sparse Retrieval☆11Mar 26, 2026Updated 7 months ago
- A high-performance constrained decoding engine based on context free grammar in Rust☆60May 22, 2025Updated last year
- A cargo subcommand to bundle a Rust project into a single file ✨☆18Jan 15, 2025Updated last year
- Nadir: Cutting-edge PyTorch optimizers for simplicity & composability! 🔥🚀💻☆15Jun 15, 2024Updated 2 years ago
- Normalize text string☆12Nov 6, 2018Updated 7 years ago
- minimalistic AI library that resembles HF's transformers☆13Dec 31, 2024Updated last year
- zero-vocab or low-vocab embeddings☆18Jul 17, 2022Updated 4 years ago
- ☆18Aug 1, 2015Updated 11 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Module support for JavaScript☆13Oct 20, 2026Updated 2 weeks ago
- High performance implementation of the WARP (SIGIR'25) retrieval engine.☆37Sep 8, 2026Updated last month
- This is a repository accompanying the survey Edge AI Meets LLM (coming soon), containing a comprehensive list of papers, codebases, toolc…☆19Jun 5, 2025Updated last year
- High-performance Machine Learning, Dynamic Auto-Differentiation and Tensor Algebra crate for Rust☆18Jun 8, 2025Updated last year
- From data to vector database effortlessly☆92May 17, 2025Updated last year
- Send line-based Neovim review comments to a live Pi session☆14Aug 25, 2026Updated 2 months ago
- A d-ary heap in Rust☆22Aug 24, 2026Updated 2 months ago
- Rust bindings for CTranslate2☆14Jun 21, 2023Updated 3 years ago
- ☆10Feb 8, 2025Updated last year
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- BLEU Score in Rust☆13Oct 1, 2026Updated last month
- In-memory vector database in Rust☆23Dec 23, 2024Updated last year
- Project for training, quantizing, and serving a static emebdding models.☆39Aug 7, 2026Updated 2 months ago
- CLI utility to inspect and explore .safetensors and .gguf files☆66Jun 20, 2026Updated 4 months ago
- Bookmarklet which pick random file containing explicit content and replaces it with something nice in your behalf.☆10Jan 7, 2023Updated 3 years ago
- Voracious radix sort☆78Dec 8, 2025Updated 10 months ago
- Llama got rusty and runs everywhere! 🦙☆54Updated this week
- Experimental compiler for deep learning models☆75Sep 18, 2025Updated last year
- 200,000+ Sentences about Donald Trump with political bias labels☆18Jun 2, 2020Updated 6 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Lexically Augmented Unified Retrieval Using Semantics☆28Updated this week
- Typescript declarations for the current live World of Warcraft Classic LUA API☆10May 7, 2023Updated 3 years ago
- Speed-of-Light SW efficiency by using ultra low-latency primitives for comms collectives☆13Apr 8, 2026Updated 6 months ago
- ☆24Jul 10, 2026Updated 3 months ago
- Datasets in the IR-Group☆12Jun 15, 2021Updated 5 years ago
- 🦞 Rust library of natural language dictionaries using character-wise double-array tries.☆39Aug 12, 2026Updated 2 months ago
- ☆17Aug 26, 2026Updated 2 months ago