Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
☆48Apr 6, 2026Updated last week
Alternatives and similar repositories for kitoken
Users that are interested in kitoken are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Anthropic MCP go implementation☆19Mar 19, 2026Updated 3 weeks ago
- PHP low-level client for Vespa. https://vespa.ai/☆17Jan 22, 2026Updated 2 months ago
- Automatically exported from code.google.com/p/esaxx☆17Jun 23, 2015Updated 10 years ago
- ☆12Apr 29, 2022Updated 3 years ago
- A high-performance constrained decoding engine based on context free grammar in Rust☆58May 22, 2025Updated 10 months ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Latent Large Language Models☆19Aug 24, 2024Updated last year
- gRPC server for hnswlib☆16Mar 6, 2023Updated 3 years ago
- Normalize text string☆12Nov 6, 2018Updated 7 years ago
- zero-vocab or low-vocab embeddings☆18Jul 17, 2022Updated 3 years ago
- minimalistic AI library that resembles HF's transformers☆13Dec 31, 2024Updated last year
- Code for the paper "Multi-Field Adaptive Retrieval," a research project on a semi-structured document retrieval☆18Feb 13, 2026Updated 2 months ago
- The omegaUp sandbox☆14Feb 13, 2023Updated 3 years ago
- ☆21Apr 16, 2024Updated last year
- Private self-improvement coaching with open-source LLMs☆17Mar 7, 2024Updated 2 years ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- AI Chat app written in GPUI and GPUI Component☆24Dec 10, 2025Updated 4 months ago
- CMS 230 - Computer Organization and Architecture☆11Sep 6, 2024Updated last year
- Model implementation for the contextual embeddings project☆47Jun 2, 2025Updated 10 months ago
- Command-line password manager☆20Mar 19, 2026Updated 3 weeks ago
- 🚨 slog: Parquet handler + Object Storage☆19Apr 2, 2026Updated last week
- Package diskcache provides a on-disk cache for storing http results.☆12Mar 27, 2026Updated 2 weeks ago
- Go package to produce a repomap based on tree-sitter☆15Jan 29, 2025Updated last year
- Shift from passive documentation to active enforcement.☆51Mar 14, 2026Updated last month
- htmx Components for ASP.NET Core☆77Updated this week
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- Website for TREC RAG☆14Aug 19, 2025Updated 7 months ago
- Repository containing code for the NAACL 2021 paper (Incorporating External Knowledge to Enhance Tabular Reasoning)☆16Jun 20, 2021Updated 4 years ago
- A Python utility for indexing file lines. Best demo honourable mention at ECIR 2024.☆23Nov 9, 2025Updated 5 months ago
- ☆14Sep 30, 2021Updated 4 years ago
- Predicting US startups survival using data science☆14Jun 5, 2020Updated 5 years ago
- Hierarchical Navigable Small World Graphs☆20Aug 17, 2024Updated last year
- 200,000+ Sentences about Donald Trump with political bias labels☆17Jun 2, 2020Updated 5 years ago
- 👑 Pytorch code for the Nero optimiser.☆21Oct 12, 2022Updated 3 years ago
- 👩 Pytorch and Jax code for the Madam optimiser.☆53Feb 9, 2021Updated 5 years ago
- Serverless GPU API endpoints on Runpod - Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- An R package to convert SingeCellExperiment and Seurat objects into anndata as comprehensively as possible.☆11Apr 23, 2025Updated 11 months ago
- Guichan is a C++ GUI library designed for games.☆14Oct 22, 2025Updated 5 months ago
- A powerful and simple asynchronous task management system that divides complex tasks into subtasks, processes them concurrently using o1 …☆16Dec 26, 2024Updated last year
- A nats micro service interacting with Ollama☆18Jun 30, 2024Updated last year
- Descriptor Vector Exchange☆76Oct 24, 2019Updated 6 years ago
- Nature's Cost Function (NCF). Finding paths of least action with gradient descent.☆18Mar 30, 2023Updated 3 years ago
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆155Oct 15, 2024Updated last year