transitive-bullshit / compare-tokenizersLinks
A test suite comparing Node.js BPE tokenizers for use with AI models.
☆31Updated 2 years ago
Alternatives and similar repositories for compare-tokenizers
Users that are interested in compare-tokenizers are comparing it to the libraries listed below
Sorting:
- Minimal and opinionated OpenAI client powered by fetch.☆195Updated last year
- OpenAI Function Calling in Typescript using Zod☆91Updated 2 years ago
- hnswlib-node provides Node.js bindings for Hnswlib☆130Updated last week
- Merge multiple streams into a unified stream☆62Updated last year
- Build your Cloudflare Workers, on Cloudflare Workers☆19Updated 3 years ago
- Split large texts into chunks with a maximum number of tokens. Split by fixed size or by sentence.☆34Updated last year
- OpenAI's tiktoken but with node bindings☆106Updated last year
- Generate vector embeddings in NodeJS☆172Updated last month
- Yet another library to extract text from MS Office and PDF files☆84Updated last month
- Refactor code, migrate to new APIs, and better understand your codebase.☆113Updated last year
- Send tRPC `subscription`s over Server-sent-events☆23Updated last year
- Load module from string using require or import.☆53Updated last year
- Super Simple API framework, type safe, automatic OpenAPI, MCP support, client RPC, streaming with SSE☆83Updated last week
- Partial JSON parser extracted from OpenAI's vendored version.☆14Updated 6 months ago
- An experimental playground starting point to build React Server applications on the Cloudflare platform using Vite.☆16Updated last year
- JSON parser for streaming objects live from an LLM's output☆36Updated last year
- support for type acquisition with url imports, npm, jsr prefixes☆16Updated last year
- Synchronize text files between browser and disk using Yjs and the File System Access API☆179Updated 3 years ago
- Use Fastify idioms for writing Cloudflare Workers and Bun servers☆101Updated last year
- Generate RSC Payloads using an LLM☆39Updated last year
- Distance functions to compare vectors☆75Updated 3 months ago
- More than Monaco, less than VSCode☆90Updated last year
- A rate-limiter using Durable Objects on CF Workers that actually doesn't rate limit anything.☆53Updated 2 years ago
- JSON in query strings, human-readable and compact☆24Updated last year
- Get detailed information about the incoming request based on the IP address.☆31Updated 8 months ago
- A VM module for Node.js that utilizes the secure environment provided by Deno☆85Updated 8 months ago
- Render Markdown from any public GitHub repo or gist☆17Updated 9 months ago
- 🪵 Fast effective logging for Cloudflare Workers☆56Updated last year
- Enable your AWS Lambda hosted applications to stream responses via Cloudflare Workers.☆22Updated 3 years ago
- Tool for patching buggy NPM packages without forking them☆29Updated 5 months ago