Alternatives and similar repositories for tokenizers

Users that are interested in tokenizers are comparing it to the libraries listed below

• Ingvarstep / GLiNER.js
  GLiNER inference in JavaScript
  ☆20Updated 7 months ago
• Lv-291 / wasm-bhtsne
  Parallel wasm Barnes-Hut t-SNE implementation written in Rust.
  ☆21Updated last year
• padolsey / xmllm
  Simple structured data from any LLM
  ☆48Updated 5 months ago
• lsb / pq.js
  Using embeddings compressed by Product Quantization, in Javascript
  ☆31Updated 2 years ago
• loro-dev / loro-examples-deno
  ☆26Updated 11 months ago
• hyparam / hyllama
  llama.cpp gguf file parser for javascript
  ☆49Updated 10 months ago
• cameronking4 / Annotated-AST-For-LLM
  Annotated AST for the purposes of LLM code repository context. Use this novel approach to pre-process your Github repository. This projec…
  ☆33Updated 10 months ago
• ShravanSunder / hnswlib-wasm
  hnswlib-wasm attempts to create a browser friendly version of hnswlib
  ☆55Updated 2 years ago
• anothersmith / node-duckdb
  DuckDB NodeJS bindings
  ☆48Updated 2 years ago
• cloudflare / workers-wonnx
  ☆46Updated 6 months ago
• dexaai / dexter
  LLM tools used in production at Dexa
  ☆104Updated 9 months ago
• marimo-team / codemirror-mcp
  CodeMirror extension to hook up a Model Context Provider (MCP)
  ☆78Updated 2 months ago
• katsumiok / ts-askit
  AskIt (for JavaScript/TypeScript): Unified programming interface for large language models (GPT-4, GPT-3.5)
  ☆34Updated 2 years ago
• xenova / model-explorer
  Browse, search, and visualize ONNX models.
  ☆34Updated 5 months ago
• lsb / sqlite-vector-search
  ☆31Updated 2 years ago
• Anush008 / fastembed-js
  Library to generate vector embeddings in NodeJS
  ☆154Updated last month
• IntrinsicLabsAI / gbnfgen
  TypeScript generator for llama.cpp Grammar directly from TypeScript interfaces
  ☆140Updated last year
• asg017 / sqlite-rembed
  A SQLite extension for generating text embeddings from remote APIs (OpenAI, Nomic, Ollama, llamafile...)
  ☆137Updated 11 months ago
• dylibso / chainsocket
  Proof of concept for a generative AI application framework powered by WebAssembly and Extism
  ☆14Updated 2 years ago
• jamsocket / jamsocket
  A monorepo for Jamsocket client libraries, the CLI, and examples
  ☆38Updated 7 months ago
• PaulKinlan / idb-vector
  A simple vector database built on idb
  ☆101Updated last year
• val-town / codemirror-codeium
  Codeium code completion integration for CodeMirror 6
  ☆82Updated last month
• mattpocock / node-DeepResearch
  Keep searching, reading webpages, reasoning until it finds the answer (or exceeding the token budget)
  ☆62Updated 7 months ago
• zya / litellmjs
  JavaScript implementation of LiteLLM.
  ☆140Updated 7 months ago
• rgbkrk / display.js
  Rich Displays for Jupyter JavaScript Kernels
  ☆25Updated 2 years ago
• hack-dance / island-ai
  ☆155Updated 4 months ago
• yoshoku / hnswlib-node
  hnswlib-node provides Node.js bindings for Hnswlib
  ☆120Updated last week
• extism / js-sdk
  Run Extism WebAssembly plug-ins / functions from JavaScript (supports major browsers, Node.js, Deno, Bun, and Cloudflare Workers)
  ☆120Updated 5 months ago
• FL33TW00D / embd
  GPU accelerated client-side embeddings for vector search, RAG etc.
  ☆65Updated last year
• RealSput / Wenode
  WebContainers, except it's a million times easier to use
  ☆88Updated 2 years ago