oramasearch / minbpeLinks
Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. Pure JavaScript port of karpathy/minbpe
☆18Updated last year
Alternatives and similar repositories for minbpe
Users that are interested in minbpe are comparing it to the libraries listed below
Sorting:
- JSON canonicalize function☆52Updated 9 months ago
- small ~1k unzip library☆16Updated 7 months ago
- Split large texts into chunks with a maximum number of tokens. Split by fixed size or by sentence.☆34Updated last year
- load a plugin via a synchronous worker☆58Updated 2 years ago
- A pair of libraries that make type checked client-server communication dramatically easier. Makes adding endpoints as easy as adding func…☆17Updated 4 years ago
- Opentelemetry instrumentation for Prisma ORM☆11Updated 4 years ago
- Peek a stream!☆13Updated 3 years ago
- 🕵️♀️ Light-weight observable implementation and functional utilities in TypeScript☆48Updated 2 years ago
- Tiny, type-safe JSON-RPC 2.0 implementation☆20Updated 2 years ago
- Yet another auto expiring, max sizable cache☆51Updated 3 months ago
- A set helpers for writing dynamic SQL queries with `pg-sql` in Javascript.☆38Updated 4 years ago
- Minify any GraphQL document in Rust☆17Updated 2 years ago
- Ponyfill and helpers for the standardized Error Causes☆53Updated last month
- Playing around with async iterators with HTTP servers☆14Updated 4 years ago
- Powerful, automatic ”watch mode” for scripts reliant on filesystem access (built on Chokidar)☆22Updated 6 months ago
- Private symbols for node.js☆14Updated 5 years ago
- A bunch of functions for building HTTP servers☆23Updated 2 months ago
- A tiny (~90B) isomorphic wrapper for `crypto.randomBytes` in Node.js and browsers.☆69Updated 2 years ago
- Generate TypeScript runtime types from your database schema.☆37Updated last year
- ☆29Updated 4 years ago
- Same as sqlite-tag but without the native sqlite3 module dependency☆19Updated 11 months ago
- ☆53Updated 4 years ago
- Alternate Promise.race() implementation which doesn't leak memory, courtesy Brian Kim (https://github.com/brainkim)☆13Updated 2 years ago
- a language acceptor for the JavaScript Pattern (regular expression internals) grammar☆16Updated 3 years ago
- A virtual Open Source project maintainer☆16Updated last year
- 👣 Effective tracing for the edge and origins☆48Updated 6 months ago
- An Opinionated Message Queue with an emitter-style API☆165Updated last week
- Benchmarking Parcel vs. Vite vs. Turbopack HMR performance☆58Updated 2 years ago
- Fast and minimal JS <head> server-side writer and client-side manager.☆27Updated last year
- Provides Lock and RwLock synchronization primitives.☆15Updated last year