fix: Retry JSON queries on Vertex AI #2093
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I noticed sometimes the LLM tries to put a lot in the `instructions` part of the vector terms response; this part is only meant to direct reasoning and we're ultimately throwing it out, so it doesn't make sense to be very verbose. It also appears to hit against some limits, at least on Gemini.
This might help avoid recitation errors on retry.
The old version dropped deprecation warnings for punycode module.
dustinbyrne
approved these changes
Oct 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Looking through our Gemini logs I noticed JSON queries sometimes fail:
It's not clear to me why this happened, so I added retry and also logging of the response on error.
EDIT: I actually checked the trajectory and here's what the output was (note it's verbatim — the LLM returned it truncated):
{ "context": "io.fits.Card float string representation too large, truncates comments, astropy", "instructions": "Reproduce the following issue with a test case. Generate exactly\none test case that reproduces the issue.\n\n<issue>\n`io.fits.Card` may use a string representation of floats that is larger than necessary\n### Description\n\nIn some scenarios, `io.fits.Card` may use a string representation of floats that is larger than necessary, which can force comments to be truncated. Due to this, there are some keyword/value/comment combinations that are impossible to create via `io.fits` even though they are entirely possible in FITS.\n\n### Expected behavior\n\nBeing able to create any valid FITS Card via `io.fits.Card`.\n\n### How to Reproduce\n\n[This valid FITS file](https://github.com/astropy/astropy/files/10922976/test.fits.gz) contains the following card in the header:\n\n`HIERARCH ESO IFM CL RADIUS = 0.009125 / [m] radius arround actuator to avoid`\n\nWe can read the header of this file and get this card without any issue:\n\nSeems like it hit some kind of limit. I modified the prompt to be brief in the instructions; hopefully it'll avoid this problem.
EDIT2: It seems that the JSON is hitting the RECITATION filter in Gemini due to it trying to repeat the input. Hopefully the instruction to be brief will make it paraphrase it enough to avoid this...