transitive-bullshit / compare-tokenizers
A test suite comparing Node.js BPE tokenizers for use with AI models.
☆30Updated last year
Alternatives and similar repositories for compare-tokenizers:
Users that are interested in compare-tokenizers are comparing it to the libraries listed below
- A Cloudflare Workers-based API for extracting and converting web page content to Markdown using DOM-Distiller and Turndown.☆21Updated this week
- Minimal and opinionated OpenAI client powered by fetch.☆188Updated 4 months ago
- support for type acquisition with url imports, npm, jsr prefixes☆16Updated 10 months ago
- Generate RSC Payloads using an LLM☆38Updated 6 months ago
- Audio Visual Playground, or Alien vs Predator? You decide...☆22Updated 6 months ago
- Basic primitives for working with binary data☆24Updated last month
- ☆13Updated 2 years ago
- Distance functions to compare vectors☆70Updated 6 months ago
- A VM module for Node.js that utilizes the secure environment provided by Deno☆77Updated last week
- A Firecrawl compatible API implemented in Cloudflare Workers☆32Updated 2 months ago
- write your request handlers once, run anywhere☆36Updated last week
- Merge multiple streams into a unified stream☆56Updated 6 months ago
- Convert Zod schemas to JSON schemas which are optionally compatible with OpenAI's structured outputs.☆17Updated 6 months ago
- OpenAI Function Calling in Typescript using Zod☆89Updated last year
- Super Simple API framework, type safe, automatic OpenAPI, MCP support, client RPC, streaming with SSE☆50Updated last week
- hnswlib-node provides Node.js bindings for Hnswlib☆109Updated this week
- An experimental playground starting point to build React Server applications on the Cloudflare platform using Vite.☆16Updated 3 months ago
- tRPC for extension and web 🧩🕸️☆36Updated 2 months ago
- Split large texts into chunks with a maximum number of tokens. Split by fixed size or by sentence.☆30Updated last year
- This Tailwind CSS plugin provides custom variants to apply styles to apps that are running on the ToDesktop platform☆18Updated last year
- GPT-2/3 tokenizer based on @latitudegames/GPT-3-Encoder that works in the browser and Deno☆32Updated last year
- A Cloudflare Page showing how to do semantic search☆49Updated 5 months ago
- Load module from string using require or import.☆51Updated 9 months ago
- OpenAI's tiktoken but with node bindings☆104Updated 11 months ago
- AI functions for Typescript☆79Updated 2 years ago
- markdown documentation language☆31Updated 6 months ago
- Build APIs and consume them in React apps with complete type safety, optimistic mutations, and offline first functionality.☆12Updated last month
- A rate-limiter using Durable Objects on CF Workers that actually doesn't rate limit anything.☆51Updated last year
- A memory efficient upload handler for hono.☆23Updated last year
- Compile Cloudflare Workers to Cap'n Proto and deliver them as minimalist Docker images.☆28Updated this week