Alternatives and similar repositories for tokenizers:

Users that are interested in tokenizers are comparing it to the libraries listed below

• FL33TW00D / embd
  GPU accelerated client-side embeddings for vector search, RAG etc.
  ☆66Updated last year
• kyegomez / Exa
  Unleash the full potential of exascale LLMs on consumer-class GPUs, proven by extensive benchmarks, with no long-term adjustments and min…
  ☆26Updated 5 months ago
• TryForefront / dataset-tool
  ☆29Updated 8 months ago
• nexusflowai / NexusBench
  Nexusflow function call, tool use, and agent benchmarks.
  ☆19Updated 4 months ago
• dexaai / dexter
  LLM tools used in production at Dexa
  ☆97Updated 3 months ago
• spirobel / bunny-llama
  iterate quickly with llama.cpp hot reloading. use the llama.cpp bindings with bun.sh
  ☆48Updated last year
• lsb / sqlite-vector-search
  ☆30Updated last year
• m1guelpf / repair-json
  Repair incomplete JSON (e.g. from streaming APIs or AI models) so it can be parsed as it's received.
  ☆33Updated last year
• cfahlgren1 / smol-copilot
  360M model running in the browser on WebGPU
  ☆21Updated 8 months ago
• cloudflare / workers-wonnx
  ☆42Updated 2 weeks ago
• Lv-291 / wasm-bhtsne
  Parallel wasm Barnes-Hut t-SNE implementation written in Rust.
  ☆18Updated 10 months ago
• fixie-ai / thefastest.ai
  Website with current metrics on the fastest AI models.
  ☆41Updated 5 months ago
• InfrHQ / Replay
  An Infr app that helps you replay & talk to everything you've ever seen.
  ☆16Updated last year
• lsb / pq.js
  Using embeddings compressed by Product Quantization, in Javascript
  ☆31Updated last year
• gitwitorg / react-eval
  Framework to evaluate LLM generated ReactJS code.
  ☆54Updated last year
• gsuuon / ad-llama
  Structured inference with Llama 2 in your browser
  ☆52Updated 5 months ago
• adrienbrault / json-schema-to-gbnf
  Converts JSON-Schema to GBNF grammar to use with llama.cpp
  ☆52Updated last year
• aiDotEngineer / aie-mcp
  MCP remote server for AI Engineer World's Fair 2025
  ☆12Updated last week
• cfahlgren1 / hf-data-explorer
  Chrome Extension for exploring Hugging Face datasets 🔎
  ☆49Updated 7 months ago
• enjalot / latent-data-modal
  Using modal.com to process FineWeb-edu data
  ☆20Updated 3 weeks ago
• dexaai / mcp-quickstart
  Setup an MCP server in 60 seconds.
  ☆12Updated 4 months ago
• loro-dev / loro-examples-deno
  ☆20Updated 5 months ago
• jungrama / syncboard
  ☆18Updated last year
• neoxelox / dspy-inspector
  DSPy program/pipeline inspector widget for Jupyter/VSCode Notebooks.
  ☆34Updated last year
• InfrHQ / desktop-client
  An Infr app that automates data collection from your PC, macOS or Linux client.
  ☆11Updated last year
• JigsawStack / omiai
  OmiAI is an opinionated AI SDK for Typescript that auto-picks the best model from a suite of curated models depending on the prompt. It i…
  ☆99Updated 2 months ago
• marimo-team / codemirror-mcp
  CodeMirror extension to hook up a Model Context Provider (MCP)
  ☆62Updated 3 months ago
• FL33TW00D / laserbeak
  Add local LLMs to your Web or Electron apps! Powered by Rust + WebGPU
  ☆102Updated last year
• bentoml / BentoVLLM
  Self-host LLMs with vLLM and BentoML
  ☆106Updated last week
• shane-kercheval / mcp-client-agent
  CLI that uses DSPy to interact with MCP servers.
  ☆17Updated last month