transitive-bullshit / compare-tokenizersLinks
A test suite comparing Node.js BPE tokenizers for use with AI models.
☆31Updated last year
Alternatives and similar repositories for compare-tokenizers
Users that are interested in compare-tokenizers are comparing it to the libraries listed below
Sorting:
- Minimal and opinionated OpenAI client powered by fetch.☆191Updated 8 months ago
- Distance functions to compare vectors☆73Updated 10 months ago
- hnswlib-node provides Node.js bindings for Hnswlib☆119Updated this week
- OpenAI Function Calling in Typescript using Zod☆91Updated last year
- Super Simple API framework, type safe, automatic OpenAPI, MCP support, client RPC, streaming with SSE☆72Updated this week
- write your request handlers once, run anywhere☆36Updated 4 months ago
- Merge multiple streams into a unified stream☆57Updated 10 months ago
- Library to generate vector embeddings in NodeJS☆142Updated 5 months ago
- Tool for patching buggy NPM packages without forking them☆27Updated last month
- Partial JSON parser extracted from OpenAI's vendored version.☆14Updated last month
- Split large texts into chunks with a maximum number of tokens. Split by fixed size or by sentence.☆34Updated last year
- Yet another library to extract text from MS Office and PDF files☆81Updated last year
- tRPC for extension and web 🧩🕸️☆43Updated 6 months ago
- A VM module for Node.js that utilizes the secure environment provided by Deno☆84Updated 4 months ago
- OpenAI's tiktoken but with node bindings☆105Updated last year
- Refactor code, migrate to new APIs, and better understand your codebase.☆113Updated last year
- support for type acquisition with url imports, npm, jsr prefixes☆16Updated last year
- Load module from string using require or import.☆52Updated last year
- A Visual Studio Code extension loads types(.d.ts) for esm.sh imports.☆37Updated 10 months ago
- Powerful machine learning library for Node.js – uses Python's scikit-learn under the hood.☆201Updated last month
- 🏜️ A sandbox that runs user-supplied JavaScript/TypeScript scripts.☆54Updated 2 years ago
- 🔂 Send patches around to keep the system in sync.☆64Updated 9 months ago
- Codeium code completion integration for CodeMirror 6☆82Updated last week
- Safe MDX React renderer that does not evaluate javascript to render. Works in Cloudflare Workers & Vercel Edge.☆82Updated 3 weeks ago
- Convert Zod schemas to JSON schemas which are optionally compatible with OpenAI's structured outputs.☆19Updated last month
- Generate RSC Payloads using an LLM☆38Updated 10 months ago
- Synchronize text files between browser and disk using Yjs and the File System Access API☆177Updated 2 years ago
- JSON in query strings, human-readable and compact☆24Updated 10 months ago
- pnpm update for pnpm workspace catalogs.☆14Updated 5 months ago
- Render Markdown from any public GitHub repo or gist☆17Updated 5 months ago