Alternatives and similar repositories for tokenizers

Users that are interested in tokenizers are comparing it to the libraries listed below

• Lv-291 / wasm-bhtsne
  Parallel wasm Barnes-Hut t-SNE implementation written in Rust.
  ☆21Updated last month
• Ingvarstep / GLiNER.js
  GLiNER inference in JavaScript
  ☆21Updated 10 months ago
• xenova / model-explorer
  Browse, search, and visualize ONNX models.
  ☆34Updated 8 months ago
• padolsey / xmllm
  Simple structured data from any LLM
  ☆52Updated 8 months ago
• hyparam / hyllama
  llama.cpp gguf file parser for javascript
  ☆50Updated last year
• Anush008 / fastembed-js
  Generate vector embeddings in NodeJS
  ☆172Updated last month
• dbSurface / dbSurface
  dbSurface is a SQL editor made for pgvector.
  ☆21Updated last month
• FurkanToprak / OkapiBM25
  Tested and profiled implementation of the OkapiBM25 algorithm. Install the npm package. Now at 130K downloads per year!
  ☆54Updated last month
• zya / litellmjs
  JavaScript implementation of LiteLLM.
  ☆145Updated 10 months ago
• dexaai / dexter
  LLM tools used in production at Dexa
  ☆106Updated last year
• frost-beta / llama3.js
  A JavaScript implementation of Llama 3 using node-mlx.
  ☆74Updated last year
• lsb / pq.js
  Using embeddings compressed by Product Quantization, in Javascript
  ☆31Updated 2 years ago
• marimo-team / codemirror-mcp
  CodeMirror extension to hook up a Model Context Provider (MCP)
  ☆79Updated last month
• nbonamy / multi-llm-ts
  A Typescript library to use LLM providers APIs in a unified way.
  ☆75Updated this week
• kyegomez / Exa
  Unleash the full potential of exascale LLMs on consumer-class GPUs, proven by extensive benchmarks, with no long-term adjustments and min…
  ☆26Updated last year
• janwilmake / universal-mcp-oauth
  Universal MCP IdP (Identity Provider) - Support Thousands of Integrations, Zero Maintenance
  ☆30Updated last month
• FL33TW00D / embd
  GPU accelerated client-side embeddings for vector search, RAG etc.
  ☆65Updated 2 years ago
• IntrinsicLabsAI / gbnfgen
  TypeScript generator for llama.cpp Grammar directly from TypeScript interfaces
  ☆141Updated last year
• RealSput / Wenode
  WebContainers, except it's a million times easier to use
  ☆92Updated 2 years ago
• lsb / sqlite-vector-search
  ☆31Updated 2 years ago
• belladoreai / llama3-tokenizer-js
  JS tokenizer for LLaMA 3 and LLaMA 3.1
  ☆117Updated 6 months ago
• yusufhilmi / client-vector-search
  A client side vector search library that can embed, store, search, and cache vectors. Works on the browser and node. It outperforms OpenA…
  ☆225Updated last year
• cloudflare / workers-wonnx
  ☆49Updated 9 months ago
• ahoylabs / gguf.js
  A Javascript library (with Typescript types) to parse metadata of GGML based GGUF files.
  ☆51Updated last year
• SilasMarvin / wasmind
  A modular framework for building massively parallel agentic systems
  ☆29Updated 4 months ago
• wizenheimer / tinylm
  Browser based ML Inference | OpenAI compliant | Run models like DeepSeek, Llama 3.2, NomicEmbed, KokoroTTS, and more
  ☆52Updated 10 months ago
• steel-dev / steel-node
  The official Node.js / Typescript library for the Steel API
  ☆14Updated last week
• cameronking4 / Annotated-AST-For-LLM
  Annotated AST for the purposes of LLM code repository context. Use this novel approach to pre-process your Github repository. This projec…
  ☆35Updated last year
• rickyrobinett / langchainjs-workers
  ☆68Updated 2 years ago
• loro-dev / loro-examples-deno
  ☆27Updated last year