Alternatives and similar repositories for tokenizers

Users that are interested in tokenizers are comparing it to the libraries listed below

• lsb / sqlite-vector-search
  ☆30Updated last year
• Ingvarstep / GLiNER.js
  GLiNER inference in JavaScript
  ☆18Updated 4 months ago
• cameronking4 / Annotated-AST-For-LLM
  Annotated AST for the purposes of LLM code repository context. Use this novel approach to pre-process your Github repository. This projec…
  ☆31Updated 6 months ago
• weaviate / typescript-embedded
  An embedded Weaviate database with TypeScript client interface
  ☆11Updated last year
• marimo-team / codemirror-mcp
  CodeMirror extension to hook up a Model Context Provider (MCP)
  ☆71Updated 5 months ago
• xenova / model-explorer
  Browse, search, and visualize ONNX models.
  ☆32Updated last month
• padolsey / xmllm
  Simple structured data from any LLM
  ☆44Updated last month
• Lv-291 / wasm-bhtsne
  Parallel wasm Barnes-Hut t-SNE implementation written in Rust.
  ☆21Updated last year
• asg017 / sqlite-rembed
  A SQLite extension for generating text embeddings from remote APIs (OpenAI, Nomic, Ollama, llamafile...)
  ☆128Updated 8 months ago
• FL33TW00D / embd
  GPU accelerated client-side embeddings for vector search, RAG etc.
  ☆66Updated last year
• cloudflare / workers-wonnx
  ☆46Updated 2 months ago
• loro-dev / loro-examples-deno
  ☆24Updated 7 months ago
• ahoylabs / gguf.js
  A Javascript library (with Typescript types) to parse metadata of GGML based GGUF files.
  ☆47Updated 11 months ago
• TryForefront / dataset-tool
  ☆31Updated 10 months ago
• baristaGeek / semantica
  Semantic Code Search Using Vectorized Abstract Syntax Trees
  ☆17Updated last year
• masylum / cheer-reader
  A port of readability.js for cheerio
  ☆25Updated 8 months ago
• maxkrieger / spaceshifter
  Finetune your embeddings in-browser
  ☆33Updated last year
• inngest / event-schemas
  Event Schema definitions for common events on the Inngest platform
  ☆16Updated last year
• dexaai / dexter
  LLM tools used in production at Dexa
  ☆100Updated 5 months ago
• Ronsor / VecDex
  Vector functions and indexing for SQLite
  ☆11Updated 2 years ago
• mandolyte / sqlitenext
  ☆14Updated 2 years ago
• Vaibhavs10 / github-issue-generator-webgpu
  ☆38Updated 7 months ago
• lsb / pq.js
  Using embeddings compressed by Product Quantization, in Javascript
  ☆31Updated last year
• epicure / llama2.js
  Inference Llama 2 in one file of pure JavaScript(HTML)
  ☆33Updated last month
• JacksonKearl / gjp-4-gpt
  JSON parser for streaming objects live from an LLM's output
  ☆34Updated last year
• InfrHQ / desktop-client
  An Infr app that automates data collection from your PC, macOS or Linux client.
  ☆11Updated last year
• dylibso / chainsocket
  Proof of concept for a generative AI application framework powered by WebAssembly and Extism
  ☆14Updated last year
• dylibso / mcpx-eval
  An open-ended eval framework for mcp.run tools
  ☆18Updated last month
• mikeal / matrika
  Next Generation Decentralized Database
  ☆15Updated 2 years ago
• sensoris / semcache
  Semantic caching layer for your LLM applications. Reuse responses and reduce token usage.
  ☆82Updated last week