Alternatives and similar repositories for tokenizers

Users that are interested in tokenizers are comparing it to the libraries listed below

• Lv-291 / wasm-bhtsne
  Parallel wasm Barnes-Hut t-SNE implementation written in Rust.
  ☆21Updated last year
• Ingvarstep / GLiNER.js
  GLiNER inference in JavaScript
  ☆20Updated 8 months ago
• xenova / model-explorer
  Browse, search, and visualize ONNX models.
  ☆34Updated 6 months ago
• FL33TW00D / embd
  GPU accelerated client-side embeddings for vector search, RAG etc.
  ☆65Updated last year
• cameronking4 / Annotated-AST-For-LLM
  Annotated AST for the purposes of LLM code repository context. Use this novel approach to pre-process your Github repository. This projec…
  ☆34Updated 11 months ago
• janwilmake / universal-mcp-oauth
  Universal MCP IdP (Identity Provider) - Support Thousands of Integrations, Zero Maintenance
  ☆29Updated last month
• lsb / pq.js
  Using embeddings compressed by Product Quantization, in Javascript
  ☆31Updated 2 years ago
• steel-dev / steel-node
  The official Node.js / Typescript library for the Steel API
  ☆14Updated this week
• rickyrobinett / langchainjs-workers
  ☆68Updated 2 years ago
• padolsey / xmllm
  Simple structured data from any LLM
  ☆49Updated 6 months ago
• TryForefront / dataset-tool
  ☆34Updated last year
• cfahlgren1 / hf-data-explorer
  Chrome Extension for exploring Hugging Face datasets 🔎
  ☆49Updated last year
• Anush008 / fastembed-js
  Library to generate vector embeddings in NodeJS
  ☆160Updated 2 months ago
• fixie-ai / llm-frameworks
  ☆19Updated last year
• PaulKinlan / idb-vector
  A simple vector database built on idb
  ☆101Updated last year
• dexaai / dexter
  LLM tools used in production at Dexa
  ☆105Updated 10 months ago
• zya / litellmjs
  JavaScript implementation of LiteLLM.
  ☆142Updated 8 months ago
• SideGuide / hacker-news-tldr
  Powered by SideGuide and GPT-3
  ☆13Updated 2 years ago
• loro-dev / loro-examples-deno
  ☆26Updated last year
• mandolyte / sqlitenext
  ☆16Updated 2 years ago
• nbonamy / multi-llm-ts
  A Typescript library to use LLM providers APIs in a unified way.
  ☆64Updated 2 weeks ago
• hyparam / hyllama
  llama.cpp gguf file parser for javascript
  ☆50Updated 11 months ago
• IntrinsicLabsAI / gbnfgen
  TypeScript generator for llama.cpp Grammar directly from TypeScript interfaces
  ☆141Updated last year
• yoshoku / hnswlib-node
  hnswlib-node provides Node.js bindings for Hnswlib
  ☆123Updated last week
• cloudflare / workers-wonnx
  ☆47Updated 7 months ago
• poloclub / mememo
  A JavaScript library that brings vector search and RAG to your browser!
  ☆156Updated last year
• Vaibhavs10 / github-issue-generator-webgpu
  ☆39Updated last year
• frost-beta / llama3.js
  A JavaScript implementation of Llama 3 using node-mlx.
  ☆73Updated last year
• mishushakov / ai-fun
  The LLM-powered function builder for TypeScript
  ☆23Updated last year
• dust-tt / srchd
  Universal agent collaboration to solve reasoning and search intensive problems
  ☆43Updated this week