num_branches missing in the code in sorting examples?? (and other questions)

[ratthachat]

Hi, thank for the great paper and code!

However, I may be missing somethings but I find few strange implementations:
For example in sorting_032.py

• in tot(), 2nd parameter is used in Generate i.e. Generate(1, 20)
• in got(), 1st parameter is used in Generate i.e. Generate(2, 1)

The 1st parameter is then used here:

class Generate(Operation)
def __init__(
        self, num_branches_prompt: int = 1, num_branches_response: int = 1
    ) 
     self.num_branches_prompt: int = num_branches_prompt

def _execute()
    prompter.generate_prompt(self.num_branches_prompt, **base_state)

BUT in
SortingPrompter.generate_prompt e.g. here
there is no used about this parameter at all. (why??)

Other questions:

• In the paper, Repeat(k) is mentioned, but there is none in the code (why?)
• In most (if not all) examples, io() and cot() are the same (why?)

Thanks again and looking forward on clarification

PS. it would be great if the authors provide a colab that is ready to run for this repository.
At the moment, it is quite difficult to quickly set up and try things.

[rgersten]

Hello ratthachatt,

thank you for looking into our work.

I am sorry for the late reply, but we were attending a conference last week.

BUT in
SortingPrompter.generate_prompt e.g. here
there is no used about this parameter at all. (why??)

The parameter num_branches_prompt is used for introducing branching inside a single prompt-response pair, i.e. asking the language model to provide num_branches_prompt thoughts/intermediate solutions in a single response, where as num_branches_response determines how often the language model is asked to provide a response for a given operation/prompt. These are different forms to implement the branching for the Generation operation.

in got(), 1st parameter is used in Generate i.e. Generate(2, 1)

You are correct, that is actually a mistake, since num_branches_prompt is not used for the generation prompts of the sorting example. The provided examples for sorting 64 and 128 elements use the correct Generate(1, 1) code for the initial splitting of the list to sort.

Thank you for noticing the mistake, which we will soon rectify.

In the paper, Repeat(k) is mentioned, but there is none in the code (why?)

The Repeat operation is indirectly implemented for example by using the num_branches_response for the Generation operation that you mention above. This parameter controls how often the language model is asked to generate an answer. Most of the operations have a similarly named parameter, that represents the Repeat operation mentioned in the paper. We introduced the Repeat operation in our research article to have a conceptually elegant solution to represent the branching, while making it also clear what is happening.

In most (if not all) examples, io() and cot() are the same (why?)

I assume that are talking about the Graph of Operations here. While this graph is the same for both operations, the used prompts differ (see examples/sorting/sorting_032.py, lines 209 to 212 for selection logic and lines 32 to 45 for IO prompt and lines 47 to 89 for the Chain of Thought prompt).

Have a nice day.

[ratthachat]

Dear @rgersten , thank you for your kind explanation!

This answer most of my questions, except one about num_branches_prompt .

I may still miss something, but could you please give me examples, where this parameter num_branches_prompt is used in your code? I search through your codes but couldn't see how it is used.

For example (as mentioned above), it looks like it is used here:

class Generate(Operation)
def __init__(
        self, num_branches_prompt: int = 1, num_branches_response: int = 1
    ) 
     self.num_branches_prompt: int = num_branches_prompt

def _execute()
    prompter.generate_prompt(self.num_branches_prompt, **base_state)

BUT if we go further to see, e.g., SortingPrompter.generate_prompt and DocMergePrompter.generate_prompt in sorting_032.py or doc_merge.py , we can see it is renamed as num_branches: int, but we cannot see thisnum_branches parameter is used at all.

Thank you again in advanced, and have a nice day too!

[rgersten]

Hello ratthachat,

I may still miss something, but could you please give me examples, where this parameter num_branches_prompt is used in your code? I search through your codes but couldn't see how it is used.

You are correct, that this feature (letting the language model do the branch inside a single prompt-answer pair) is currently not used in any of the provided use cases. However it is an option for the user to employ this parameter similar to how, for example, the inputs and their lengths are used as parameters in the merge prompt in the sorting use case. Please have a look as reference at lines 178 to 183 for the setting of the parameters and lines 139 to 155 for the merge prompt in the source code of the sorting use case for 32 elements: https://github.com/spcl/graph-of-thoughts/blob/main/examples/sorting/sorting_032.py

We initially experimented with that parameter, however ultimately decided to query the LLM for different answers using the API instead, since it requires less parsing and therefore avoids related errors.

Thank you again in advanced, and have a nice day too!

You are welcome and thank you.

Have a nice day.

[ratthachat]

Thanks again @rgersten , now I better understand the logic.
As a last request to conclude this issue, provided your mention

We initially experimented with that parameter, however ultimately decided to query the LLM for different answers using the API instead, since it requires less parsing and therefore avoids related errors.

could you please give some code example (e.g. the one you made experiments) on
Prompter.generate_prompt
if num_branches_prompt would be used?

I just want to make sure I understand your design correctly and concretely.

[rgersten]

Hello ratthachat,

could you please give some code example (e.g. the one you made experiments) on
Prompter.generate_prompt
if num_branches_prompt would be used?

I found the following example for one of our old experiments:

def generate_prompt(
    self, num_branches: int, current_state: str, original: str, **kwargs
) -> str:
...
num_next_steps = (
    "Possible next steps:\n"
    if num_branches < 1
    else f"Provide {num_branches} possible next steps:\n"
)
...

I hope that example will help with the understanding of the num_branches parameter.

Have a nice day.