docs: update server streaming mode documentation

[CentricStorm]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Server documentation:

• Mentioned that streaming mode uses a different response format
• Added a link to documentation of that response format
• Reduced the size of n_predict in existing non-streamed example script (because on some computers 512 tokens can take a long time)
• Made style of existing non-streamed example script more consistent

[CentricStorm]

Updated the streaming mode example script with split data handling, which has been tested with these unit tests:

// Chunks with a total of 9 tokens: " token token token token, token, token,", split at different positions.
const chunks = [
    `data: {"co`,
    `ntent":" token"}\n\n`,
    `data: {"content":" token"}\n`,
    `\n`,
    `data: {"content":" token"}\n\n`,
    `data: {"content":" token"}\n\ndata: {"co`,
    `ntent":","}\n\n`,
    `data: {"content":" token"}\n\ndata: {"content":","}\n`,
    `\n`,
    `data: {"content":" token"}\n\ndata: {"content":","}\n\n`
]

Avoided using Node.js readline so that it can work in browsers as well with minimal modification.

[ngxson]

Btw have you been able to test it with latest version on master branch? We added [DONE] as last data chunk at some point, to be aligned with openai implementation

[CentricStorm]

Btw have you been able to test it with latest version on master branch? We added [DONE] as last data chunk at some point, to be aligned with openai implementation

It seems like #9459 only added the [DONE] event for the OpenAI-compatible /chat/completions API, not for the /completion API that this example uses:

https://github.com/ggerganov/llama.cpp/blob/678d7994f4da0af3d29046be99950ac999ee9762/examples/server/server.cpp#L3027

[CentricStorm]

Example script still works with b4291 (ce8784b), but changed localhost to 127.0.0.1 because of a separate issue with the latest version of Node.js.

[ngxson]

On second thought, I think it's not a good idea to add this to our documentation. Because we already follow SSE standard (except for the POST method), client code should be trivial to implement.

The documentation should be reserved for things that can only be found in llama.cpp and not on the internet.

In this case, the code you provided is the same as openai implementation (because they also use SSE+POST method), there are many libraries on npm that can handle this (for example, this). So adding this here brings no more additional info to the docs, while adding maintenance cost in the future.

[CentricStorm]

On second thought, I think it's not a good idea to add this to our documentation. Because we already follow SSE standard (except for the POST method), client code should be trivial to implement.

The documentation should be reserved for things that can only be found in llama.cpp and not on the internet.

Removed example code.

[CentricStorm]

Suggestions implemented.