Replies: 6 comments 1 reply
-
An example where code
error stack trace
Did I miss any settings to get this to work? I feel like this example should be manageable. |
Beta Was this translation helpful? Give feedback.
-
Also experiencing this issue and seems to happen at random? My assumption is it has to do with how Instructor forms there prompts and if there are any retries involved. |
Beta Was this translation helpful? Give feedback.
-
Hey man, if you're using the OpenAI compatible client, we don't do anything with the prompts, all we do is convert your schema to a OpenAI Compatible Schema and from there send it together with the request. We do append an error message with the generated content when a validation error is detected though. The validation errors itself seems to be thrown because the model isn't repairing what was originally generated in a prior request. You can experiment with higher retries and see if that works! Out of curiosity, what model are you using with this? |
Beta Was this translation helpful? Give feedback.
-
llama3.1 I'm thinking of something like tracking back the current token if the current output can't be extended to match the format, and then backing off to the 2nd most likely token, then 3rd etc. If I understand it correctly, llama.cpp already supports this? (even more general as you can supply any BFN grammar) |
Beta Was this translation helpful? Give feedback.
-
We are using gpt4o model from OpenAI. Will try the higher retries method
…On Sat, Aug 31, 2024 at 3:05 AM Ivan Leo ***@***.***> wrote:
Hey man, if you're using the OpenAI compatible client, we don't do
anything with the prompts, all we do is convert your schema to a OpenAI
Compatible Schema and from there send it together with the request. We do
append an error message with the generated content when a validation error
is detected though.
The validation errors itself seems to be thrown because the model isn't
repairing what was originally generated in a prior request. You can
experiment with higher retries and see if that works!
Out of curiosity, what model are you using with this?
—
Reply to this email directly, view it on GitHub
<#952 (comment)>
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AOG52IDGLRFSYMFBNLYLESTZUFTLPBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVE3DKMZVHA4TCMBSQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGQ4DKNBZGM4TGNFHORZGSZ3HMVZKMY3SMVQXIZI>
.
You are receiving this email because you commented on the thread.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>
.
|
Beta Was this translation helpful? Give feedback.
-
Higher retries don't fix the problem if you're using 0 temperature and in some cases (in my case) temperature 0 is required or desirable. Is there a way to specify retry behavior? For example, I would like to adjust original the prompt on failure by adding the missing field error message to it. I couldn't find a way to do that in the docs. |
Beta Was this translation helpful? Give feedback.
-
When I first looked into this project, I thought you could get a guarantee (not just reasonable high chance) that you will get a response in the required schema.
From my experience, this isn't the case, unless I used the library incorrectly.
I think this could be implemented by using a fallback in the stream, i.e. when you receive a token that is incompatible with the schema, you instead go on with the 2nd most likely option, etc.
How feasible is this idea? For GPT-4, you have access to the logprobs, which would enable this approach, right?
Beta Was this translation helpful? Give feedback.
All reactions