mvitlov / tiktoken

tiktoken is a BPE tokeniser for use with OpenAI's models
25Updated last year

Alternatives and similar repositories for tiktoken:

Users that are interested in tiktoken are comparing it to the libraries listed below