transitive-bullshit / compare-tokenizersLinks
A test suite comparing Node.js BPE tokenizers for use with AI models.
☆31Updated last year
Alternatives and similar repositories for compare-tokenizers
Users that are interested in compare-tokenizers are comparing it to the libraries listed below
Sorting:
- Minimal and opinionated OpenAI client powered by fetch.☆192Updated 9 months ago
- OpenAI Function Calling in Typescript using Zod☆91Updated last year
- hnswlib-node provides Node.js bindings for Hnswlib☆119Updated last week
- Distance functions to compare vectors☆74Updated 11 months ago
- Super Simple API framework, type safe, automatic OpenAPI, MCP support, client RPC, streaming with SSE☆73Updated this week
- OpenAI's tiktoken but with node bindings☆105Updated last year
- Yet another library to extract text from MS Office and PDF files☆81Updated last year
- Library to generate vector embeddings in NodeJS☆150Updated 3 weeks ago
- Split large texts into chunks with a maximum number of tokens. Split by fixed size or by sentence.☆34Updated last year
- 🏜️ A sandbox that runs user-supplied JavaScript/TypeScript scripts.☆55Updated 2 years ago
- A VM module for Node.js that utilizes the secure environment provided by Deno☆84Updated 5 months ago
- Build your Cloudflare Workers, on Cloudflare Workers☆19Updated 2 years ago
- support for type acquisition with url imports, npm, jsr prefixes☆15Updated last year
- Codeium code completion integration for CodeMirror 6☆82Updated 3 weeks ago
- A Cloudflare Page showing how to do semantic search☆50Updated 10 months ago
- ☆129Updated 2 years ago
- Partial JSON parser extracted from OpenAI's vendored version.☆14Updated 2 months ago
- JSON parser for streaming objects live from an LLM's output☆35Updated last year
- Merge multiple streams into a unified stream☆58Updated 11 months ago
- ☆22Updated 2 months ago
- Automagically generates summaries from html or text.☆67Updated 2 years ago
- Get detailed information about the incoming request based on the IP address.☆31Updated 4 months ago
- Powerful machine learning library for Node.js – uses Python's scikit-learn under the hood.☆201Updated 2 months ago
- write your request handlers once, run anywhere☆36Updated 5 months ago
- Node.js bindings for faiss☆136Updated last year
- This Tailwind CSS plugin provides custom variants to apply styles to apps that are running on the ToDesktop platform☆18Updated last year
- Enable your AWS Lambda hosted applications to stream responses via Cloudflare Workers.☆23Updated 2 years ago
- Send tRPC `subscription`s over Server-sent-events☆23Updated last year
- Synchronize text files between browser and disk using Yjs and the File System Access API☆177Updated 2 years ago
- Basic Cloudflare Pages template☆16Updated last year