facebookresearch / Exact-Byte-Level-Probabilities-from-Tokenized-LMsLinks

Example implementation of "Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles" by Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen UllrichWork conducted as part of a MetaAI internship.
13Updated 7 months ago

Alternatives and similar repositories for Exact-Byte-Level-Probabilities-from-Tokenized-LMs

Users that are interested in Exact-Byte-Level-Probabilities-from-Tokenized-LMs are comparing it to the libraries listed below

Sorting: