Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆54May 10, 2026Updated last month
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Anthropic MCP go implementation☆19May 29, 2026Updated 2 weeks ago
- PHP low-level client for Vespa. https://vespa.ai/☆17Jan 22, 2026Updated 4 months ago
- Automatically exported from code.google.com/p/esaxx☆17Jun 23, 2015Updated 10 years ago
- Latent Large Language Models☆19Aug 24, 2024Updated last year
- Resa: Transparent Reasoning Models via SAEs☆49Sep 23, 2025Updated 8 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- gRPC server for hnswlib☆16Mar 6, 2023Updated 3 years ago
- zero-vocab or low-vocab embeddings☆18Jul 17, 2022Updated 3 years ago
- minimalistic AI library that resembles HF's transformers☆13Dec 31, 2024Updated last year
- Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks☆17Jan 15, 2025Updated last year
- ☆29May 14, 2025Updated last year
- 🧬🔍 Vecgo is a pure Go, embeddable, hybrid vector database designed for high-performance production workloads. It combines commit-orient…☆19Jan 19, 2026Updated 4 months ago
- Private self-improvement coaching with open-source LLMs☆17Mar 7, 2024Updated 2 years ago
- CMS 230 - Computer Organization and Architecture☆10Sep 6, 2024Updated last year
- Model implementation for the contextual embeddings project☆47Jun 2, 2025Updated last year
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- Deterministic agent harness for Temporal (in Rust)☆37Updated this week
- PostHog with text analytics extensions, serving as an advanced LLM analytics platform.☆15Sep 17, 2024Updated last year
- Music structure analysis with community detection methods☆18Oct 24, 2019Updated 6 years ago
- A CLI tool for running AI agents inside microVM sandboxes☆45May 26, 2026Updated 2 weeks ago
- Create a palette of N colors or convert True Color images to indexed ones. Includes png2gpl and png2act.☆18May 29, 2026Updated 2 weeks ago
- 🚨 slog: Parquet handler + Object Storage☆19Jun 3, 2026Updated last week
- Go package to produce a repomap based on tree-sitter☆15Jan 29, 2025Updated last year
- Website for TREC RAG☆14May 30, 2026Updated 2 weeks ago
- Command-line password manager☆23Jun 1, 2026Updated last week
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- A Python utility for indexing file lines. Best demo honourable mention at ECIR 2024.☆23Nov 9, 2025Updated 7 months ago
- Predicting US startups survival using data science☆15Jun 5, 2020Updated 6 years ago
- Website for Applied-LLMs work☆29May 5, 2026Updated last month
- Hierarchical Navigable Small World Graphs☆24Aug 17, 2024Updated last year
- Connect Client SDK and CLI☆19Mar 11, 2026Updated 3 months ago
- 🤖 AI is with you.☆14Nov 14, 2024Updated last year
- An R package to convert SingeCellExperiment and Seurat objects into anndata as comprehensively as possible.☆14May 19, 2026Updated 3 weeks ago
- ✂️ OpenAI's tiktoken tokenizer written in Go☆20Jan 31, 2025Updated last year
- htmx Components for ASP.NET Core☆96Jun 7, 2026Updated last week
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- A powerful and simple asynchronous task management system that divides complex tasks into subtasks, processes them concurrently using o1 …☆16Dec 26, 2024Updated last year
- Guichan is a C++ GUI library designed for games.☆14Oct 22, 2025Updated 7 months ago
- A tiny utility to help save you a lot of effort with long winded `#[cfg()]` checks in Rust.☆97Apr 16, 2025Updated last year
- Export specific notes to general md for static site generation, such as Hexo, Hugo, or Astro☆35Nov 19, 2025Updated 6 months ago
- a lightweight bpmn workflow engine☆26Feb 26, 2026Updated 3 months ago
- Authentication Callout Library☆23Jun 2, 2026Updated last week
- Tulp is a command-line tool that can help you create and process piped content using the power of ChatGPT directly from the terminal.☆21Jan 20, 2026Updated 4 months ago