microsoft / sarathi-serve

A low-latency & high-throughput serving engine for LLMs
231Updated last month

Related projects

Alternatives and complementary repositories for sarathi-serve