Repair Prompt Improvements

[Stevenic]

I wanted to share some observations about the current repair prompt. It's too passive. You'll get more reliable repairs if your repair prompt is presented as a set of very specific instructions for what changes to make to the response, especially with gpt-3.5-turbo. I originally took the same approach of just showing the model the validator errors in my AlphaWave project but I found that it occasionally would fail to repair its response. I re-wrote all of my repair feedback to be targeted repair instructions and I have yet to see it not repair a response. Sometimes it takes 2 calls but it always repairs it.

It's not enough to just update your prompt envelop you need to actually re-write all of your validation errors to be repair instructions. So instead of saying "The 'foo.bar' property is missing" you should say "Add a 'foo.bar' property to your response". You can read through the details of what I needed to do to make OpenAI Functions reliable in this medium post.

[steveluc]

Thanks! We are aware that we can get much better on repair, and really appreciate the specific information about your experiences. We are waiting on changes to this feature as we gain more experience with large, production applications and their use of TypeChat. So far, we've been able to use the current lightweight approach to repair but no doubt we'll be needing to beef it up in response to feeback from bringing up large applications.

[Stevenic]

I definitely feel like you're on the right path with TypeChat just sharing some observations I've had about how to improve the reliability of the models output. I firmly believe you can get all of these modern models to 99%+ reliability with proper feedback and further fine tuning...