microsoft / sarathi-serve

A low-latency & high-throughput serving engine for LLMs
245Updated 2 months ago

Related projects

Alternatives and complementary repositories for sarathi-serve