microsoft / sarathi-serve

A low-latency & high-throughput serving engine for LLMs
319Updated last month

Alternatives and similar repositories for sarathi-serve:

Users that are interested in sarathi-serve are comparing it to the libraries listed below