Alternatives and similar repositories for tokenizers:

Users that are interested in tokenizers are comparing it to the libraries listed below

• FL33TW00D / embd
  GPU accelerated client-side embeddings for vector search, RAG etc.
  ☆66Updated last year
• dexaai / dexter
  LLM tools used in production at Dexa
  ☆93Updated 2 months ago
• m1guelpf / repair-json
  Repair incomplete JSON (e.g. from streaming APIs or AI models) so it can be parsed as it's received.
  ☆33Updated last year
• katsumiok / ts-askit
  AskIt (for JavaScript/TypeScript): Unified programming interface for large language models (GPT-4, GPT-3.5)
  ☆34Updated last year
• asg017 / sqlite-rembed
  A SQLite extension for generating text embeddings from remote APIs (OpenAI, Nomic, Ollama, llamafile...)
  ☆112Updated 4 months ago
• dylibso / chainsocket
  Proof of concept for a generative AI application framework powered by WebAssembly and Extism
  ☆14Updated last year
• FL33TW00D / laserbeak
  Add local LLMs to your Web or Electron apps! Powered by Rust + WebGPU
  ☆102Updated last year
• transitive-bullshit / tavily
  SDK for the Tavily search API which is tailored for LLM agents.
  ☆12Updated 9 months ago
• ahoylabs / gguf.js
  A Javascript library (with Typescript types) to parse metadata of GGML based GGUF files.
  ☆47Updated 7 months ago
• RealSput / Wenode
  WebContainers, except it's a million times easier to use
  ☆68Updated 2 years ago
• adlumal / GoalChain
  GoalChain for goal-orientated LLM conversation flows
  ☆68Updated 3 months ago
• IntrinsicLabsAI / gbnfgen
  TypeScript generator for llama.cpp Grammar directly from TypeScript interfaces
  ☆136Updated 8 months ago
• weaviate / typescript-embedded
  An embedded Weaviate database with TypeScript client interface
  ☆11Updated 8 months ago
• gsuuon / ad-llama
  Structured inference with Llama 2 in your browser
  ☆52Updated 4 months ago
• Yonom / assistant-ui-artifacts
  ☆15Updated 7 months ago
• themaximalist / llm.js
  Simple LLM library for JavaScript
  ☆43Updated last month
• token-js / token.js
  Integrate 200+ LLMs with one TypeScript SDK using OpenAI's format.
  ☆187Updated last week
• Vaibhavs10 / github-issue-generator-webgpu
  ☆36Updated 3 months ago
• try-reason / reason
  RΞASON is a minimalistic Typescript framework for building great LLM apps
  ☆49Updated 2 months ago
• zya / litellmjs
  JavaScript implementation of LiteLLM.
  ☆115Updated last month
• sugarcane-ai / sugarcane-ai
  npm like package ecosystem for Prompts 🤖
  ☆49Updated last month
• Lv-291 / wasm-bhtsne
  Parallel wasm Barnes-Hut t-SNE implementation written in Rust.
  ☆17Updated 9 months ago
• TryForefront / dataset-tool
  ☆27Updated 6 months ago
• dmarcos / llama2.c-web
  Simple repo that compiles and runs llama2.c on the Web
  ☆53Updated last year
• SciPhi-AI / r2r-js
  js client for R2R: production-ready RAG engine with a sh*t ton of features.
  ☆12Updated 6 months ago
• lspahija / force-multiplier
  Use AI to edit your documents in real-time. Provide feedback and let the AI do all the work.
  ☆26Updated 7 months ago
• h2oai / enterprise-h2ogpte
  Client Code Examples, Use Cases and Benchmarks for Enterprise h2oGPTe RAG-Based GenAI Platform
  ☆83Updated last month
• Anush008 / fastembed-js
  Library to generate vector embeddings in NodeJS
  ☆107Updated 3 months ago