Increase temperature on retries of test file and code file searches

[kgilpin]

Consider the two search logs for retries of choose-test-files. They are essentially identical.

search.log
search.log

Therefore, there is not much likelihood of obtaining a different result on retry.

Introduce a temperature change on retry, to increase the likelihood of getting different results.

astropy__astropy-8872.zip

[github-actions]

Title: Implement Temperature Adjustment for Retries in Test and Code File Searches

Problem: The current implementation of the choose_test_files and choose_code_files functions performs multiple search attempts, but with identical parameters, resulting in repetitive and redundant outcomes across retries. To increase the likelihood of different results upon retries, the search mechanism needs a mechanism like temperature adjustment.

Analysis: The problem arises from the deterministic nature of repeated search attempts using identical parameters. These searches are likely based on some form of heuristic or probabilistic matching, for which introducing a "temperature" parameter could modulate the randomness or exploration factor involved. The goal of a temperature parameter is to allow for more exploratory searches on retries, rather than optimizing for precision. This is a common technique in probabilistic models to increase the variety of outputs.

Incorporating a temperature parameter in retry attempts necessitates a systematic approach to increase the variability of the search results. The temperature in computational search contexts is a hyperparameter that can control the randomness of search output, and a higher temperature generally results in more random (and thus diverse) search results. Gradually increasing the temperature parameter with each retry can introduce the desired variability.

The likely components needing modification are the search methods in the choose_test_files and choose_code_files functions. These functions utilize an Editor class for searching, which would need to be adjusted to accept a variable temperature parameter.

Proposed Changes:

1. solver/workflow/choose_test_file.py: Modify the function that initiates and manages the search process. Add a mechanism to incrementally increase the temperature parameter with each retry attempt. Pass this dynamic temperature to the Editor.search method.

2. solver/workflow/choose_code_files.py: Similarly, modify the function here to adjust the temperature setting for retry attempts dynamically, ensuring it uses a progressively higher temperature parameter for each subsequent search retry.

3. solver/tests/workflow/test_choose_test_file.py: Update test cases for the choose_test_files functionality to account for the new temperature parameter, ensuring that the modified searches are capturing scenarios with different temperatures and validating against expected results.

4. solver/tests/workflow/test_choose_code_files.py: As above, ensure the test setup and assertions capture temperature adjustments and validate the range of search outputs.

5. solver/workflow/editor.py or equivalent utility: If relied upon, the Editor class or equivalent should be adapted to include a temperature parameter in its search logic, ensuring that it can interpret and apply this parameter to modify search randomness.

By implementing these changes, the enhanced search logic will vary with retries, leading to a broader exploration of potential results, thus increasing the potential of obtaining different, possibly more suitable files upon retries.

[dividedmind]

Why would the log files be different? AFAICT they only show the input, not the output. Presumably the input will always be the same. (BTW, are we not passing the project tree to choose files? Seems like it would be very useful.)

[kgilpin]

You're right, the search logs are not that helpful in this case. However, the issue remains - the LLM tends to repeat the same proposed solutions over and over with little variation. Fixing #88 may help.

Do you think a bit of temperature would be useful on retries?

Why would the project tree be useful to choose files? The LLM is getting RAG results of snippets, these have an implied file hierarchy that it can see.

[dividedmind]

Do you think a bit of temperature would be useful on retries?

Yes, I think it's a good idea.

Why would the project tree be useful to choose files? The LLM is getting RAG results of snippets, these have an implied file hierarchy that it can see.

I don't know, I'm just thinking on how I would approach a problem like that: I'd definitely look at git ls-files in addition to git grep. You might be right that it won't in fact improve over just the search, but I think it's worth a try; in particular, it will allow the llm to suggest files that don't come up in the search for whatever reason or are culled because of context size.