Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆55May 10, 2026Updated last month
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Anthropic MCP go implementation☆19May 29, 2026Updated last month
- Automatically exported from code.google.com/p/esaxx☆17Jun 23, 2015Updated 11 years ago
- Trainable embedding transformation for confidence estimation, feature extraction, explainability and conversion from dense to sparse.☆28Jun 23, 2026Updated last week
- A high-performance constrained decoding engine based on context free grammar in Rust☆59May 22, 2025Updated last year
- Latent Large Language Models☆19Aug 24, 2024Updated last year
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Resa: Transparent Reasoning Models via SAEs☆49Sep 23, 2025Updated 9 months ago
- gRPC server for hnswlib☆16Mar 6, 2023Updated 3 years ago
- A cargo subcommand to bundle a Rust project into a single file ✨☆18Jan 15, 2025Updated last year
- A PHP library for writing, reading, and validating llms.txt Markdown files.☆27Jun 1, 2026Updated last month
- Minimidi is a minimal header-only MIDI file library.☆15Apr 4, 2026Updated 3 months ago
- Normalize text string☆12Nov 6, 2018Updated 7 years ago
- zero-vocab or low-vocab embeddings☆18Jul 17, 2022Updated 3 years ago
- ☆29May 14, 2025Updated last year
- A GPUI based diff viewer and Codex orchestrator for vibe engineering☆57May 26, 2026Updated last month
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- The omegaUp sandbox☆15Feb 13, 2023Updated 3 years ago
- ☆21Apr 16, 2024Updated 2 years ago
- 🧬🔍 Vecgo is a pure Go, embeddable, hybrid vector database designed for high-performance production workloads. It combines commit-orient…☆19Jan 19, 2026Updated 5 months ago
- Private self-improvement coaching with open-source LLMs☆18Mar 7, 2024Updated 2 years ago
- AI Chat app written in GPUI and GPUI Component☆27Dec 10, 2025Updated 6 months ago
- CMS 230 - Computer Organization and Architecture☆10Sep 6, 2024Updated last year
- Application configuration and scripts for search on https://docs.vespa.ai/☆12Updated this week
- Model implementation for the contextual embeddings project☆47Jun 2, 2025Updated last year
- PostHog with text analytics extensions, serving as an advanced LLM analytics platform.☆15Sep 17, 2024Updated last year
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Serve HTTP on a tailnet☆22Oct 31, 2024Updated last year
- Music structure analysis with community detection methods☆18Oct 24, 2019Updated 6 years ago
- A CLI tool for running AI agents inside microVM sandboxes☆46May 26, 2026Updated last month
- Kafka Connect Vespa sink connector☆17Apr 17, 2025Updated last year
- Package diskcache provides a on-disk cache for storing http results.☆12Mar 27, 2026Updated 3 months ago
- Go package to produce a repomap based on tree-sitter☆15Jan 29, 2025Updated last year
- Website for TREC RAG☆14Jun 17, 2026Updated 2 weeks ago
- Hierarchical Navigable Small World Graphs☆25Aug 17, 2024Updated last year
- Connect Client SDK and CLI☆19Updated this week
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ✂️ OpenAI's tiktoken tokenizer written in Go☆20Jan 31, 2025Updated last year
- A powerful and simple asynchronous task management system that divides complex tasks into subtasks, processes them concurrently using o1 …☆15Dec 26, 2024Updated last year
- A nats micro service interacting with Ollama☆18Jun 30, 2024Updated 2 years ago
- Guichan is a C++ GUI library designed for games.☆15Oct 22, 2025Updated 8 months ago
- A tiny utility to help save you a lot of effort with long winded `#[cfg()]` checks in Rust.☆97Apr 16, 2025Updated last year
- A safer, drop-in replacement for Go's syscall/js JavaScript package.☆19Mar 5, 2023Updated 3 years ago
- Tulp is a command-line tool that can help you create and process piped content using the power of ChatGPT directly from the terminal.☆22Jan 20, 2026Updated 5 months ago