perk11 / large-model-proxy

Large Model Proxy is designed to make it easy to run multiple resource-heavy Large Models (LM) on the same machine with limited amount of VRAM/other resources. It listens on a dedicated port for each proxied LM, making them always available to the clients connecting to these ports.
45Updated last month

Related projects

Alternatives and complementary repositories for large-model-proxy