niieani / gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.
508Updated last month

Alternatives and similar repositories for gpt-tokenizer:

Users that are interested in gpt-tokenizer are comparing it to the libraries listed below