Why can't multiple apis be triggered at the same time

[zhengzhanpeng]

This is an area that has been a headache for me, and I hope someone can answer me

[tranhoangnguyen03]

Do you mean concurrent requests?

[zpzheng]

Do you mean concurrent requests?

yes

[abetlen]

@zhengzhanpeng this currently under development in #771

[isaac-chung]

@abetlen I noticed that llama.cpp's server supports concurrent requests and continuous batching as well https://github.com/ggerganov/llama.cpp/tree/master/examples/server. To enable that for this library, would it be as straightforward as exposing the relevant command line options? Or am I missing something obvious?