Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle omitted commas in parser #1265

Open
anerli opened this issue Dec 20, 2024 · 0 comments
Open

Handle omitted commas in parser #1265

anerli opened this issue Dec 20, 2024 · 0 comments

Comments

@anerli
Copy link

anerli commented Dec 20, 2024

LLM can occasionally respond with JSON that omits commas. Handling this in the parser to recover from this issue would prevent this edge case from failing.

Example:

 {
      "scene": "The browser displays a detailed product page for the Nike V2K Run women's shoes. The main image shows a beige/tan colored shoe with a chunky sole and Nike swoosh. To the left, there's a column of smaller images showing different angles of the shoe. On the right side of the page, we see the product name 'Nike V2K Run', category 'Women's Shoes', and pricing information ($102.97, discounted from $120, 14% off). There's also a note about using code LASTSHOT for an extra 25% off. Below this information, there's a grid of color options, showing 14 different colorways for this shoe model. Each color option is represented by a small image of the shoe in that particular color. At the bottom, there's a 'Select Size' label, indicating where size options would be displayed. The page also includes standard Nike website navigation elements at the top, including categories like New, Men, Women, Kids, Jordan, and Sale."
    
      "reasoning": "We have successfully completed the previous step by selecting a shoe with multiple color choices - the Nike V2K Run, which has 14 color options as clearly displayed on this product page. This satisfies our requirement for 'Select any shoe with multiple color choices'. The current step has been completed, and we can now move on to the next step in the test case, which is to change the shoe color. However, our current task is simply to confirm that we have completed the step of selecting a shoe with multiple color choices, which we have done.",
    
      "action": "COMPLETE"
    }

Here, Claude (Sonnet 3.5) omitted a comma after the scene value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant