Guideline Decomposition

[mc-dorzo]

Motivation

Today, guidelines are specified via the API (and CLI) with two parts: a condition and an action.

However, in real-world use cases, we often find that people like to specify composite conditions and actions. For example:

• Condition: The customer wants to make a transaction and has already specified to whom as well as the desired amount but has not yet confirmed their PIN code
• Action: Ask for their PIN code and confirm it with the system

While there may be different ways to approach guideline design, this looks like something that should be allowed and expected to work well.

However, we know from experience that the attention of LLMs struggle with following instructions. Currently, the GuidelineProposer and MessageEventGenerator overcome this challenge by looking at each guideline in isolation—as we know, this dramatically improves the instruction-following consistency.

But such composite conditions and actions kind of "backdoor" this modeling, and break it to some extent, in that consistency is not quite the same in such cases.

Solution Proposal

While keeping guideline API the same (a singly-specified condition/action), decompose guidelines into one or more conditions and actions. The example above would thus be rendered as follows:

• Conditions: 1) The customer wants to make a transaction ; 2) The customer has specified to whom they wish to make the transfer ; 3) The customer has specified the desired amount of the transaction ; 4) The customer has not yet confirmed their PIN code
• Actions: 1) Ask for the customer's PIN code ; 2) Confirm the customer's PIN code with the system

Given such decomposition, we refactor GuidelineProposer to evaluate the conditions in isolation, thus achieving more control and clearer optimization opportunities, hence higher accuracy.

Discussion

1. How do we keep runtime complexity down, as far as possible?
2. How (and when) exactly do we perform the decomposition, and are there any edge cases to be aware of?
3. Are there seemingly decomposable cases that are actually inherently atomic? If so, how do we identify them?
4. Whatever solution we come up with, how do we approach (inevitable) issues? The solution should lend itself to enabling clear strategies such that, "When X goes wrong, you should do Y to remedy it."

[vahuja4]

Folks, it is time to sleep and this is the first thing that has come to me. How about we fine-tune a GliNER based model to recognize condition spans and action spans. Then, we feed the original sentence along with the detected spans, and ask the LLM to give us the condition, action pairs.
https://huggingface.co/knowledgator

[MCBarKar]

So for example, if we focus only on the 'condition' part, you're suggesting finetuning GliNER to recognize entities of type 'condition', such that when we give it the input "The customer wants to make a transaction and has already specified to whom as well as the desired amount but has not yet confirmed their PIN code" with a single label ["condition"], it would provide us with the spans ['The customer wants to make a transaction', 'has already specified to whom', 'as well as the the desired amount', 'has not yet confirmed their PIN code']. And then an LLM could build 4 new conditions from these spans, such that each condition is understandable on its own.

Did I understand it correctly?

[MCBarKar]

Correct me if I'm wrong here - but NER can only return exact quotes from the input as entities, so we would have to pass it through an LLM to get conditions that are understandable by themselves.

It's not too bad here because this mechanism would need to run only when you insert a new guidleline, so it doesn't effect the latency customers experience.

[kichanyurd]

@MCBarKar @vahuja4 but if we have an LLM request anyway during indexing, I ask myself if we'd get better performance by asking it to directly break down the conditions into standalone parts. Also I suspect that, if we indeed do it during indexing, the LLM request would be negligible in the grand scheme of things. What do you guys think?