transitive-bullshit / compare-tokenizersLinks
A test suite comparing Node.js BPE tokenizers for use with AI models.
☆31Updated last year
Alternatives and similar repositories for compare-tokenizers
Users that are interested in compare-tokenizers are comparing it to the libraries listed below
Sorting:
- Minimal and opinionated OpenAI client powered by fetch.☆191Updated 7 months ago
- hnswlib-node provides Node.js bindings for Hnswlib☆115Updated last week
- Distance functions to compare vectors☆73Updated 10 months ago
- OpenAI's tiktoken but with node bindings☆105Updated last year
- Load module from string using require or import.☆52Updated last year
- Super Simple API framework, type safe, automatic OpenAPI, MCP support, client RPC, streaming with SSE☆67Updated last week
- OpenAI Function Calling in Typescript using Zod☆90Updated last year
- Convert Zod schemas to JSON schemas which are optionally compatible with OpenAI's structured outputs.☆18Updated last month
- support for type acquisition with url imports, npm, jsr prefixes☆16Updated last year
- Library to generate vector embeddings in NodeJS☆139Updated 4 months ago
- Hot module reloading for nodejs☆112Updated 6 months ago
- Node.js bindings for faiss☆134Updated last year
- Build your Cloudflare Workers, on Cloudflare Workers☆19Updated 2 years ago
- Merge multiple streams into a unified stream☆57Updated 9 months ago
- Enable your AWS Lambda hosted applications to stream responses via Cloudflare Workers.☆23Updated 2 years ago
- ☆50Updated 4 months ago
- ☆67Updated 2 years ago
- An experimental playground starting point to build React Server applications on the Cloudflare platform using Vite.☆16Updated 7 months ago
- ☆130Updated 2 years ago
- Refactor code, migrate to new APIs, and better understand your codebase.☆113Updated 11 months ago
- Split large texts into chunks with a maximum number of tokens. Split by fixed size or by sentence.☆34Updated last year
- AI functions for Typescript☆79Updated 2 years ago
- Serialize live running code to Javascript☆47Updated 8 months ago
- A VM module for Node.js that utilizes the secure environment provided by Deno☆84Updated 3 months ago
- GPT-2/3 tokenizer based on @latitudegames/GPT-3-Encoder that works in the browser and Deno☆32Updated last year
- 🏜️ A sandbox that runs user-supplied JavaScript/TypeScript scripts.☆54Updated 2 years ago
- Yet another library to extract text from MS Office and PDF files☆81Updated last year
- Continue block comments in CodeMirror☆20Updated last week
- A Cloudflare Page showing how to do semantic search☆50Updated 8 months ago
- JSON parser for streaming objects live from an LLM's output☆34Updated last year