Alternatives and similar repositories for tokenizers

Users that are interested in tokenizers are comparing it to the libraries listed below

• lsb / pq.js
  Using embeddings compressed by Product Quantization, in Javascript
  ☆31Updated 2 years ago
• Ingvarstep / GLiNER.js
  GLiNER inference in JavaScript
  ☆19Updated 6 months ago
• hyparam / hyllama
  llama.cpp gguf file parser for javascript
  ☆45Updated 9 months ago
• padolsey / xmllm
  Simple structured data from any LLM
  ☆47Updated 4 months ago
• Lv-291 / wasm-bhtsne
  Parallel wasm Barnes-Hut t-SNE implementation written in Rust.
  ☆21Updated last year
• FL33TW00D / embd
  GPU accelerated client-side embeddings for vector search, RAG etc.
  ☆65Updated last year
• spirobel / bunny-llama
  iterate quickly with llama.cpp hot reloading. use the llama.cpp bindings with bun.sh
  ☆51Updated last year
• xenova / model-explorer
  Browse, search, and visualize ONNX models.
  ☆34Updated 4 months ago
• dexaai / dexter
  LLM tools used in production at Dexa
  ☆103Updated 8 months ago
• marimo-team / codemirror-mcp
  CodeMirror extension to hook up a Model Context Provider (MCP)
  ☆76Updated last month
• loro-dev / loro-examples-deno
  ☆26Updated 10 months ago
• lsb / sqlite-vector-search
  ☆31Updated 2 years ago
• zya / litellmjs
  JavaScript implementation of LiteLLM.
  ☆139Updated 6 months ago
• FurkanToprak / OkapiBM25
  Tested and profiled implementation of the OkapiBM25 algorithm. Install the npm package. Now at 45K downloads per year!
  ☆42Updated 3 months ago
• asg017 / sqlite-rembed
  A SQLite extension for generating text embeddings from remote APIs (OpenAI, Nomic, Ollama, llamafile...)
  ☆135Updated 10 months ago
• drivly / ai
  Agentic Workflow Platform to do Business-as-Code and deliver valuable Services-as-Software through simple APIs and SDKs
  ☆32Updated last week
• cloudflare / workers-wonnx
  ☆46Updated 5 months ago
• poloclub / mememo
  A JavaScript library that brings vector search and RAG to your browser!
  ☆142Updated last year
• ahoylabs / gguf.js
  A Javascript library (with Typescript types) to parse metadata of GGML based GGUF files.
  ☆51Updated last year
• Anush008 / fastembed-js
  Library to generate vector embeddings in NodeJS
  ☆145Updated last week
• janwilmake / universal-mcp-oauth
  Universal MCP IdP (Identity Provider) - Support Thousands of Integrations, Zero Maintenance
  ☆26Updated this week
• cfahlgren1 / hf-data-explorer
  Chrome Extension for exploring Hugging Face datasets 🔎
  ☆49Updated last year
• IntrinsicLabsAI / gbnfgen
  TypeScript generator for llama.cpp Grammar directly from TypeScript interfaces
  ☆140Updated last year
• frost-beta / llama3.js
  A JavaScript implementation of Llama 3 using node-mlx.
  ☆73Updated last year
• gsuuon / ad-llama
  Structured inference with Llama 2 in your browser
  ☆53Updated 10 months ago
• cameronking4 / Annotated-AST-For-LLM
  Annotated AST for the purposes of LLM code repository context. Use this novel approach to pre-process your Github repository. This projec…
  ☆33Updated 9 months ago
• zuplo / api-key-manager
  ☆27Updated 2 weeks ago
• katsumiok / ts-askit
  AskIt (for JavaScript/TypeScript): Unified programming interface for large language models (GPT-4, GPT-3.5)
  ☆34Updated last year
• cztomsik / ggml-js
  JavaScript bindings for the ggml-js library
  ☆44Updated 6 months ago
• kyegomez / Exa
  Unleash the full potential of exascale LLMs on consumer-class GPUs, proven by extensive benchmarks, with no long-term adjustments and min…
  ☆26Updated 10 months ago