Support stream response

[guoqingbao]

Open this issue to track the progress of stream response feature in candle-vllm.

[guoqingbao]

Current progress:

66 tokens/s on A100 for LLaMa2 7B (BF16)

Note: there is a problem for candle-vllm release version on certain environments (Rust tokio runtime cpu usage abnormal), tring to fix. Use the debug version instead at the moment.

[guoqingbao]

The stream generation hang has been addressed, refer to #42 Candle-vllm can now generate 71 tokens/s per request on A100 for LLaMa2 7B (BF16) in release mode, which is very close to vLLM (Pytorch backend) (72 tokens/s).

[guoqingbao]

TODO: batched streaming

[EricLBuehler]

TODO: batched streaming

I think the only thing holding up batched streaming is the scheduler, right? Is there anything else that you can think of which would need to change?

[guoqingbao]

TODO: batched streaming

I think the only thing holding up batched streaming is the scheduler, right? Is there anything else that you can think of which would need to change?

I think the client (ChatUI) may also need to be revised to support batched streaming. Currently, the frontend and backend (candle-vllm) are both designed to process a single query (batch size = 1) at a time. For example, the chat request and completion messages derived from the OpenAI API were designed on the basis of a single-user response (history messages of a given user). These need to be revised to handle messages from multiple users. A unique user ID is thus needed to indicate which messages belong to which user. The backend service will then accept chat completion requests that contain history messages from multiple users, stack and pad them into tensorized input tokens (batch size > 1). The backend also needs to respond with decoding results according to the finished status, e.g., in a single stacked request, some user messages are finished (removed from streaming pipeline) while others are still in decoding (streaming).

While there is another strategy that does not require changes to the client side. The backend service can accept chat completion requests from different users, and a message queue and a standalone backend thread can be used to grab multiple completion requests from the queue and process them in batches. The current strategy of candle-vllm for processing chat completion requests is to process the current request while blocking incoming requests (model.lock, pipeline.lock) until the current request is finished.

[guoqingbao]

TODO: batched streaming

This have been supported in #69

[guoqingbao]

Streaming response and batched streaming response have been supported.