Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆61Aug 10, 2026Updated last month
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Fast Tokens☆147Updated this week
- The TypeScript API client for Zoo.☆16Updated this week
- A Rust crate offering similar functionality to the Python transformers package using Candle.☆15Nov 19, 2024Updated last year
- Anthropic MCP go implementation☆20Updated this week
- PHP low-level client for Vespa. https://vespa.ai/☆17Jan 22, 2026Updated 8 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Run all the tests at the same time with modal.com☆11Mar 2, 2024Updated 2 years ago
- Build script crate to run npm commands.☆20Nov 16, 2022Updated 3 years ago
- A high-performance constrained decoding engine based on context free grammar in Rust☆59May 22, 2025Updated last year
- Latent Large Language Models☆19Aug 24, 2024Updated 2 years ago
- A cargo subcommand to bundle a Rust project into a single file ✨☆18Jan 15, 2025Updated last year
- (NeurIPS 2025) Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instruction☆20Nov 2, 2025Updated 10 months ago
- Multiplayer terminal multiplexer where multiple users and machines can connect to a same session☆72Updated this week
- ☆18Aug 1, 2015Updated 11 years ago
- An implementation of random number generator based on rdrand instruction☆27May 15, 2026Updated 4 months ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Normalize text string☆12Nov 6, 2018Updated 7 years ago
- zero-vocab or low-vocab embeddings☆18Jul 17, 2022Updated 4 years ago
- High performance implementation of the WARP (SIGIR'25) retrieval engine.☆36Sep 8, 2026Updated 2 weeks ago
- Cross-platform GPU LLM inference in Rust☆20Jun 24, 2026Updated 2 months ago
- CLI utility to inspect and explore .safetensors and .gguf files☆64Jun 20, 2026Updated 3 months ago
- Lexical Augmented Unified Retrieval Using Semantics☆28Updated this week
- Teaching a humanoid to walk(ish), then displaying in your browser (using tensorflow.js and reinforcement learning)☆10Sep 7, 2020Updated 6 years ago
- Experimental compiler for deep learning models☆75Sep 18, 2025Updated last year
- ⚡Japanese sentence splitting(日本語文境界判定器), 40–250× faster via a Rust-accelerated Python library with near-perfect API compatibility with …☆77Oct 14, 2025Updated 11 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Rust bindings for CTranslate2☆14Jun 21, 2023Updated 3 years ago
- A Tile-based 2D rendering engine.☆12Jun 17, 2019Updated 7 years ago
- fc is a research-grade, lossless floating point compressor.☆76Apr 30, 2026Updated 4 months ago
- In-memory vector database in Rust☆23Dec 23, 2024Updated last year
- ☆40Sep 15, 2026Updated last week
- Pure Java Llama2 inference with optional multi-GPU CUDA implementation☆15Sep 2, 2023Updated 3 years ago
- Rust port of C++ llvm::APFloat library☆23Updated this week
- Application configuration and scripts for search on https://docs.vespa.ai/☆12Updated this week
- A program for querying and modifying information in a datalog-like format backed by SQLite.☆40Feb 17, 2022Updated 4 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Full Text Search Benchmark, a tool for comparing and evaluating full-text search engines.☆26Jul 14, 2026Updated 2 months ago
- Super-fast Structured Outputs☆874Sep 4, 2026Updated 2 weeks ago
- Rust Solana IDL types definitions de/serializable with serde☆18Jul 17, 2024Updated 2 years ago
- Repository for the Q-Filters method (https://arxiv.org/pdf/2503.02812)☆35Mar 7, 2025Updated last year
- build your own vector database -- the littlest hnsw☆74Jan 7, 2025Updated last year
- [COLM 2026] Resa: Transparent Reasoning Models via SAEs☆49Sep 23, 2025Updated last year
- Single header, 100 line implementation of 32bit PCG random number generator. Extremely fast RNG with simple API☆14Dec 11, 2019Updated 6 years ago