Implement retry mechanism for LLM failures

[gentlementlegen]

          I would like to do that, but the endpoint is changeable so if another API is used (I know that shiv recommended others) the return code of the API can differ (and actually the return code of OpenAPI doesn't indicate that the comment is too long as far as I remember, just a failure) and parsing the error message feels unreliable.

See error code 429: https://platform.openai.com/docs/guides/error-codes

Originally posted by @gentlementlegen in #225 (comment)

Description

Currently if the LLM fails, we throw and abort the run completely, which cancels all the rewards and results, and needs a new run to be triggered afterwards. While some errors are not recoverable, the ones like us sending too many tokens to be handled can be avoided by splitting the prompt and retrying with a smaller prompt until it fits with the given model.

Needed changes

We should implement:

• a retry mechanism which, on errors that are not fatal, would give another try with the LLM
• covering cases like the token amount sent being too large, responses having truncated JSON content, network failures
• post a message saying the results are being retried
• set a failure limit
• throw in the end if every try is unsecessfull
• add related tests

[ubiquity-os-beta]

Note

The following contributors may be suitable for this task:

gentlementlegen

78% Match ubiquity-os-marketplace/text-conversation-rewards#4

[whilefoo]

/start

[ubiquity-os-beta]

Beneficiary

0xA0B11F474d8ECE1205d38c66d5F2bE8917675d60

Tip

• Use /wallet 0x0000...0000 if you want to update your registered payment wallet address.
• Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
• Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the task.