oramasearch / minbpe
Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. Pure JavaScript port of karpathy/minbpe
☆17Updated 11 months ago
Alternatives and similar repositories for minbpe:
Users that are interested in minbpe are comparing it to the libraries listed below
- A p2p version-controlled repo (built on hypercore)☆11Updated 3 years ago
- Peek a stream!☆14Updated 2 years ago
- Private symbols for node.js☆14Updated 4 years ago
- Generate TypeScript runtime types from your database schema.☆36Updated 6 months ago
- Type-safe SQL builder using ES2015 template tags☆11Updated 3 years ago
- Determines whether a Node file is a Module (`import`) or a Script (`require`)☆11Updated 4 years ago
- Working group notes for graphql-js☆27Updated this week
- A fast fifo implementation similar to the one powering nextTick in Node.js core☆62Updated 8 months ago
- Opentelemetry instrumentation for Prisma ORM☆11Updated 3 years ago
- Fast Map built for keys that are always fixed size uniformly distributed buffers.☆39Updated 2 years ago
- Running Node within Node (a fork of synchronous-worker)☆27Updated last year
- load a plugin via a synchronous worker☆56Updated last year
- Easy, transparent ”watch mode ” for filesystem access (powered by Chokidar)☆20Updated last month
- ☆12Updated 5 years ago
- relay with the relay. apollo without the apollo. suspense without the... suspense. fetch then render via async generators.☆22Updated last year
- JSON canonicalize function☆46Updated last year
- Same as sqlite-tag but without the native sqlite3 module dependency☆19Updated last year
- 📋 A polyfill for URLPattern for web-browsers, older versions of deno and non-unstable deno☆11Updated 3 years ago
- NPR allows you to run your CLI dependencies without having a copy in node_modules.☆16Updated 2 years ago
- Draft plugin of the HTTP 103 implementation☆22Updated 7 months ago
- Automatically federate a Mercurius GraphQL service☆32Updated 3 weeks ago
- Dockerfile-js, render well-formed Dockerfiles.☆19Updated last year
- A set helpers for writing dynamic SQL queries with `pg-sql` in Javascript.☆38Updated 3 years ago
- Speed up zlib operations by running them using the sync APIs but in a Worker☆29Updated 4 years ago
- Optimize the performance of important tasks by delaying background-tasks☆21Updated 4 years ago
- The one-stop shop solution for JS Proxies and FFI APIs.☆25Updated 4 months ago
- Nanoid generate with custom dictionaries☆14Updated 3 years ago
- Tiny, type-safe JSON-RPC 2.0 implementation☆20Updated last year
- Split large texts into chunks with a maximum number of tokens. Split by fixed size or by sentence.☆28Updated 11 months ago
- Enables `attributechanged` element's listener.☆15Updated 4 years ago