Structured JSON output key ordering issue causes incorrect reasoning steps

[yuan-li]

Description of the bug:

I’ve encountered a bug when using the structured output feature with the Gemini model. Specifically, I noticed that json keys in the returned response appear sorted alphabetically rather than in the order defined by my provided schema. This seems to interfere with the chain-of-thought reasoning steps.

Actual vs expected behavior:

I just copied the example from OpenAI (https://platform.openai.com/docs/guides/structured-outputs#chain-of-thought)

from pydantic import BaseModel

class Step(BaseModel):
    explanation: str
    output: str

class MathReasoning(BaseModel):
    steps: list[Step]
    final_answer: str

model = genai.GenerativeModel("gemini-1.5-flash-8b")
result = model.generate_content(
    "how can I solve 8x + 7 = -23",
    generation_config=genai.GenerationConfig(
        response_mime_type="application/json", response_schema=MathReasoning, temperature=0
    ),
)

The response is

{'final_answer': '-5',
 'steps': [{'explanation': 'Subtract 7 from both sides of the equation to isolate the term with x.',
   'output': '8x = -30'},
  {'explanation': 'Divide both sides of the equation by 8 to solve for x.',
   'output': 'x = -30/8'},
  {'explanation': 'Simplify the fraction.', 'output': 'x = -15/4'}]}

Here, the final answer appears before steps, which disrupted the reasoning steps and resulted in an incorrect final answer. It implies that the keys might be sorted alphabetically, so a workaround would be like this:

from pydantic import BaseModel

class Step(BaseModel):
    explanation: str
    output: str

class MathReasoning(BaseModel):
    calculation_steps: list[Step]
    final_answer: str

model = genai.GenerativeModel("gemini-1.5-flash-8b")
result = model.generate_content(
    "how can I solve 8x + 7 = -23",
    generation_config=genai.GenerationConfig(
        response_mime_type="application/json", response_schema=MathReasoning, temperature=0
    ),
)

And the response is as expected this time:

{'calculation_steps': [{'explanation': 'Subtract 7 from both sides of the equation.',
   'output': '8x + 7 - 7 = -23 - 7'},
  {'explanation': 'Simplify both sides of the equation.',
   'output': '8x = -30'},
  {'explanation': 'Divide both sides of the equation by 8.',
   'output': '8x / 8 = -30 / 8'},
  {'explanation': 'Simplify both sides of the equation.',
   'output': 'x = -30/8'},
  {'explanation': 'Simplify the fraction.', 'output': 'x = -15/4'}],
 'final_answer': '-15/4'}

Any other information you'd like to share?

This behavior suggests that the keys may be sorted alphabetically internally, rather than following the schema order. It would be helpful if the model could preserve the original field order to maintain the intended reasoning flow.

[Giom-V]

Hello @yuan-li,
Thank you for the feedback. I'm routing it internally to the folk in charge of structured output.

[yuan-li]

Thank you @Giom-V

[seniorb]

I'm having this issue as well, where the json is output with alphabetically arranged keys and not in the order specified

[Giom-V]

The feature is still in the backlog, but in the meantime, have you tried the thinking model?

[benhylak]

Having this exact issue, and it's not alphabetical. It's a pre-defined, seemingly random order. It's a very bad bug. Like the original post said, it makes it impossible to enforce a specific reasoning flow.

[karayanni]

@yuan-li - does the problem persist even if you use a hard coded Schema instead of a Pydantic Models? might be a temporary by-pass

We had different but similar problem that was solved by using a schema and adding a required attribute

Example with schema and required attribute

[frankang]

@karayanni This problem persists even if I use a required attribute field. The response did not follow the order specified in that field.