Add o1 support with structured output

[CTY-git]

PR Checklist

• The commit message follows our guidelines: Code of conduct
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• Does this PR introduce a breaking change?
• Include PR in release notes?

PR Type

• Bugfix
• Feature
• Refactoring
• Build /CI
• Documentation
• Others

What is the current behavior?

Currently o1 models are not supported due to outdated tiktoken version. Current tiktoken version is 0.7.0
In addition, o1 models do not support structured outputs.
Issue Number: N/A

What is the new behavior?

I have bumped our own fork of code2prompt to depend on version ^0.8.0 of tiktoken which adds encoding mapping of o1 models.

I have opted to always follow the recommendation here when structured output is required of o1 models.

Other information

Copilot reviewed 6 out of 6 changed files in this pull request and generated no suggestions.

Comments skipped due to low confidence (3)

patchwork/common/client/llm/openai_.py:32

• The line contains a tab character which should be removed.

gpt-4	

patchwork/common/client/llm/openai_.py:115

• The line is unreachable due to the preceding return statement.

return self.client.chat.completions.create(**NotGiven.remove_not_given(input_kwargs))

patchwork/common/client/llm/openai_.py:177

• The string concatenation should use f-string formatting for consistency.

"content": f"Given the following data, format it with the given response format: {o1_choice.message.content}"

[patched-codes]

File Changed: <UNKNOWN>

Details: The code introduces special handling for models starting with "o1" (such as o1-preview, o1-mini) when JSON output is required. This change implements a two-step process for JSON formatting, which is not natively supported by these models.

Affected Code Snippet:

def __o1_chat_completion(
    self,
    model: str,
    messages: List[Dict[str, str]],
    *,
    temperature: float,
    top_p: float,
    n: int,
    stream: bool,
    stop: Optional[Union[str, List[str]]],
    max_tokens: Optional[int],
    presence_penalty: float,
    frequency_penalty: float,
    logit_bias: Optional[Dict[str, float]],
    user: Optional[str],
    response_format: Optional[Dict[str, str]],
    seed: Optional[int],
    **kwargs: Any,
) -> ChatCompletion:
    # Implementation details...

Start Line: N/A

End Line: N/A

---

Details: The code modifies the last message by appending schema information to the prompt when the response_format type is "json_schema". This ensures that the o1 model receives the necessary information to generate a response compatible with the desired JSON schema.

Affected Code Snippet:

if response_format["type"] == "json_schema":
    messages[-1]["content"] += f"\n\nYou must respond according to this JSON Schema:\n{response_format['schema']}"

Start Line: N/A

End Line: N/A

---

Details: The implementation uses a two-step process where the initial response from the o1 model is post-processed using the "gpt-4o-mini" model to ensure proper JSON formatting. This approach maintains compatibility with existing API patterns while adding specialized handling for o1 models.

Affected Code Snippet:

completion = self.create_completion(
    model="gpt-4o-mini",
    prompt=f"Parse the following into valid JSON:\n\n{choice['message']['content']}",
    max_tokens=8000,
    temperature=0.0,
)
parsed_content = completion.choices[0].text.strip()
choice["message"]["content"] = parsed_content

Start Line: N/A

End Line: N/A

---

Details: The code properly accumulates token usage from both the initial completion and parsing steps, ensuring accurate reporting of resource usage for these specialized JSON formatting requests.

Affected Code Snippet:

usage["completion_tokens"] += completion.usage.total_tokens
usage["total_tokens"] += completion.usage.total_tokens

Start Line: N/A

End Line: N/A