transitive-bullshit / compare-tokenizersLinks
A test suite comparing Node.js BPE tokenizers for use with AI models.
☆31Updated 2 years ago
Alternatives and similar repositories for compare-tokenizers
Users that are interested in compare-tokenizers are comparing it to the libraries listed below
Sorting:
- Minimal and opinionated OpenAI client powered by fetch.☆194Updated 11 months ago
- OpenAI Function Calling in Typescript using Zod☆91Updated last year
- hnswlib-node provides Node.js bindings for Hnswlib☆125Updated 2 weeks ago
- Merge multiple streams into a unified stream☆62Updated last year
- Distance functions to compare vectors☆75Updated last month
- Yet another library to extract text from MS Office and PDF files☆84Updated last year
- 🏜️ A sandbox that runs user-supplied JavaScript/TypeScript scripts.☆55Updated 2 years ago
- OpenAI's tiktoken but with node bindings☆106Updated last year
- Super Simple API framework, type safe, automatic OpenAPI, MCP support, client RPC, streaming with SSE☆79Updated last month
- support for type acquisition with url imports, npm, jsr prefixes☆16Updated last year
- Use Fastify idioms for writing Cloudflare Workers and Bun servers☆101Updated last year
- Tool for patching buggy NPM packages without forking them☆29Updated 4 months ago
- Library to generate vector embeddings in NodeJS☆163Updated 2 months ago
- Partial JSON parser extracted from OpenAI's vendored version.☆14Updated 4 months ago
- Build your Cloudflare Workers, on Cloudflare Workers☆19Updated 3 years ago
- AI functions for Typescript☆79Updated 2 years ago
- ☆49Updated 8 months ago
- This Tailwind CSS plugin provides custom variants to apply styles to apps that are running on the ToDesktop platform☆18Updated last year
- Generate RSC Payloads using an LLM☆38Updated last year
- Split large texts into chunks with a maximum number of tokens. Split by fixed size or by sentence.☆34Updated last year
- Codeium code completion integration for CodeMirror 6☆83Updated 2 months ago
- A Visual Studio Code extension loads types(.d.ts) for esm.sh imports.☆42Updated last year
- Load module from string using require or import.☆53Updated last year
- tRPC for extension and web 🧩🕸️☆46Updated 9 months ago
- JSON parser for streaming objects live from an LLM's output☆36Updated last year
- ☆26Updated 10 months ago
- 💬 Estimate the number of tokens an OpenAI chat completion will use☆91Updated 3 weeks ago
- Enable your AWS Lambda hosted applications to stream responses via Cloudflare Workers.☆23Updated 2 years ago
- Refactor code, migrate to new APIs, and better understand your codebase.☆113Updated last year
- A tinny(~3kb) utility that can simplify cross window(iframes, workers, same origin windows) and electron main / renderer process communic…☆13Updated last month