Has anyone seen GPT-4o-Mini failing to properly fill out an Instructor model?

[faroukcharkas]

My Setup
Using TOOLS as my mode, 0.2 for my temperature, and 4096 for my max_tokens. Using the asynchronous OpenAI client. My prompt token size is around 10,000 (a lot, I know), and I'm using a model that has the following fields:

1. bool
2. str
3. List of Enums (20 Enum options, average List length of 5)
4. 2 datetime's
5. 2 int's

Observations

• gpt-3.5-turbo and gpt-4o work perfectly
• Short prompts (like a paragraph or so)

[jxnl]

Short suggestion is to use Literal["A", "B"] vs Enum. enum performance is so bad i almost want to warn users code...

[faroukcharkas]

• Tried with using a Literal with 20 options, then 5 options.
• Tried with removing the List and just asking it to choose one

Both were unsuccessful, it seems like the only thing that helped was reducing the prompt size. Strange that it still works pretty reliably with gpt-3.5-turbo though.