James-QiuHaoran / LLM-serving-with-proxy-modelsLinks

Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction | A tiny BERT model can tell you the verbosity of an LLM (with low latency overhead!)
35Updated last year

Alternatives and similar repositories for LLM-serving-with-proxy-models

Users that are interested in LLM-serving-with-proxy-models are comparing it to the libraries listed below

Sorting: