What need to be done to enable automatic function calling & "auto" tool choice?

[samuelint]

I would like to use functions calling & stream=True
But I have error Automatic streaming tool choice is not supported

---

From here:

llama-cpp-python/llama_cpp/llama_chat_format.py

Line 3751 in 816d491

raise ValueError("Automatic streaming tool choice is not supported")

What need to be done to support "auto" tool choice with streaming?
Why it was not implemented at first? Why is it not supported? @abetlen

I can try to implement it, be would be useful to have history of the feature, so if it's not possible I'll not lose my time.

Thanks !

[yamikumo-DSD]

This is my assumption on it.

llama-cpp-python's function call seems to force LLM to output json format and them parse it.
You notice combination of streaming and json parser is really tricky when you try to implement it by yourself.
Consider following incomplete output of json schema

{
    "arg_1": "value 1",
    "arg_2": "val

The LLM is appropriately trying to obey the rule, but this intermediate state is still not valid as function calling nor normal texts.
If stream=False this is not troublesome since only matured outputs is gonna be fed into the parser. However, when stream=True, this immature outputs are gonna be exposed on user interface, which must be weird.

What we can do to avoid it is to auto-complete the LLM's outputs (e.g. adding closure sequence ""\n}" to the above example)
I think it's tough to implement decent auto-completion for json, tbh.
So, I decided to write XML function calling pipeline on my own.

[samuelint]

If this is the only problem, I think it's easily solvable.
There's even library like https://pypi.org/project/json-stream/ to iteratively parse a json.

[lsorber]

@samuelint Are you still interested in giving this a go? We ran into this limitation with RAGLite as well. Streaming tool_choice="auto" would be an awesome improvement to llama-cpp-python!