Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support images directly in UserMessage #387

Merged
merged 48 commits into from
Jan 6, 2025
Merged
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
bc1e7cf
Move OpenaiChatModel UserImageMessage support into openai_chat_model.py
jackmpcollins Aug 13, 2024
27370b1
Implement message_to_anthropic_message for UserImageMessage
jackmpcollins Aug 13, 2024
6f4ae58
Get test passing using _combine_messages
jackmpcollins Aug 13, 2024
2fbd2b2
WIP: Add ImageBytes, ImageUrl. Expand UserMessage types.
jackmpcollins Aug 13, 2024
1eb89b5
Merge branch 'main' into allow-images-in-user-message
jackmpcollins Dec 3, 2024
186876d
Remove duplicate message_to_x_message for UserImageMessage
jackmpcollins Dec 3, 2024
f225d1a
Add make test-fix-snapshots
jackmpcollins Dec 3, 2024
925c20b
Move image_bytes fixtures into top-level conftest
jackmpcollins Dec 3, 2024
0c7f307
Fix typing for ImageBytes.mime_type. Add tests
jackmpcollins Dec 3, 2024
6c4d872
Fix mypy errors on UserMessage conversion typing
jackmpcollins Dec 3, 2024
eba51a9
Validate bytes are valid image
jackmpcollins Dec 3, 2024
3fa37c9
Use ImageBytes in UserImageMessage conversion functions
jackmpcollins Dec 3, 2024
a8dbb40
Add typevar UserMessageContentT
jackmpcollins Dec 3, 2024
b0c97c3
Fix mypy errors due to UserMessage now generic
jackmpcollins Dec 3, 2024
882b238
Attempt to coerce type in Placeholder.format
jackmpcollins Dec 4, 2024
59249bd
Fix: list -> Iterable in UserMessage serialization
jackmpcollins Dec 4, 2024
31a47f6
Fix: return message in message_to_openai_message
jackmpcollins Dec 4, 2024
e07f364
Merge branch 'main' into allow-images-in-user-message
jackmpcollins Jan 1, 2025
df575d4
Make Placeholder a BaseModel
jackmpcollins Jan 1, 2025
ce1334f
Make ContentT covariant
jackmpcollins Jan 1, 2025
9ddd4cd
Add covariant PlaceholderT
jackmpcollins Jan 1, 2025
dfc8a93
Fix type checking for UserMessage format
jackmpcollins Jan 1, 2025
87b81ca
Remove unused type ignores
jackmpcollins Jan 1, 2025
026e436
Require pydantic 2.10 to fix generic in BaseModel
jackmpcollins Jan 1, 2025
c8c7bfa
Revert "Remove unused type ignores"
jackmpcollins Jan 1, 2025
71e411e
Remove pydantic url from error messages in tests
jackmpcollins Jan 1, 2025
20e9146
Add TypeAlias UserMessageContentBlock
jackmpcollins Jan 1, 2025
19346e6
Add trailing .0 to pydantic version in pyproject
jackmpcollins Jan 1, 2025
d2e9b80
Use TypeAdapter for Placeholder coercion
jackmpcollins Jan 1, 2025
2cb7379
Add typing-extensions as dependency
jackmpcollins Jan 1, 2025
34cf578
Remove todo for testing AssistantMessage with FunctionCall
jackmpcollins Jan 1, 2025
17e4558
Ignore logfire not configured warnings
jackmpcollins Jan 1, 2025
fd636b0
Deprecate UserImageMessage
jackmpcollins Jan 1, 2025
0be8bf9
Add make test-snapshots-create and improve naming
jackmpcollins Jan 2, 2025
cf8a55c
Add tests for UserMessage with ImageBytes/Url
jackmpcollins Jan 2, 2025
f35fdf1
Upgrade mypy version
jackmpcollins Jan 2, 2025
6f42619
Add tests for ImageBytes with ChatModel
jackmpcollins Jan 2, 2025
75df0a8
Improve AssistantMessage typing, add tests
jackmpcollins Jan 2, 2025
26f3282
Handle Literal string in AssistantMessage.format typing
jackmpcollins Jan 2, 2025
5bf4412
Add NotPlaceholder Protocol
jackmpcollins Jan 2, 2025
ccb30c0
Fix type hints for UserMessage
jackmpcollins Jan 5, 2025
8e04974
Add github issue link for failing type tests
jackmpcollins Jan 5, 2025
a6ad13c
Improve handling of Literal in AssistantMessage typing
jackmpcollins Jan 6, 2025
0855b8a
Rename to PlaceholderTypeT
jackmpcollins Jan 6, 2025
f5477bd
Remove done todo re name not in kwargs error
jackmpcollins Jan 6, 2025
5012ad2
Add top-level imports for ImageBytes, ImageUrl
jackmpcollins Jan 6, 2025
225479d
Update docs for vision
jackmpcollins Jan 6, 2025
b443e09
Add note about Placeholder coercion
jackmpcollins Jan 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove pydantic url from error messages in tests
jackmpcollins committed Jan 1, 2025
commit 71e411e9b8b2282d89b9dc6271e113d392b5c9fc
Original file line number Diff line number Diff line change
@@ -41,28 +41,28 @@ interactions:
uri: https://api.openai.com/v1/chat/completions
response:
body:
string: 'data: {"id":"chatcmpl-AXM5kjh54ArE89qKZKNsCik5EZIEt","object":"chat.completion.chunk","created":1732513108,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"index":0,"id":"call_6Dk7MgfrU5OMIvVdcxiJ9TLE","type":"function","function":{"name":"return_country","arguments":""}}],"refusal":null},"logprobs":null,"finish_reason":null}],"usage":null}
string: 'data: {"id":"chatcmpl-Al1DBOr3iDv5vEmEq6YxCSZ31FFeO","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"index":0,"id":"call_xnsDVVMJ4rh23jW9WEC22IyZ","type":"function","function":{"name":"return_country","arguments":""}}],"refusal":null},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5kjh54ArE89qKZKNsCik5EZIEt","object":"chat.completion.chunk","created":1732513108,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBOr3iDv5vEmEq6YxCSZ31FFeO","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5kjh54ArE89qKZKNsCik5EZIEt","object":"chat.completion.chunk","created":1732513108,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"name"}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBOr3iDv5vEmEq6YxCSZ31FFeO","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"name"}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5kjh54ArE89qKZKNsCik5EZIEt","object":"chat.completion.chunk","created":1732513108,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\""}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBOr3iDv5vEmEq6YxCSZ31FFeO","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\""}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5kjh54ArE89qKZKNsCik5EZIEt","object":"chat.completion.chunk","created":1732513108,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"Australia"}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBOr3iDv5vEmEq6YxCSZ31FFeO","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"Canada"}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5kjh54ArE89qKZKNsCik5EZIEt","object":"chat.completion.chunk","created":1732513108,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBOr3iDv5vEmEq6YxCSZ31FFeO","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5kjh54ArE89qKZKNsCik5EZIEt","object":"chat.completion.chunk","created":1732513108,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null}
data: {"id":"chatcmpl-Al1DBOr3iDv5vEmEq6YxCSZ31FFeO","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null}


data: {"id":"chatcmpl-AXM5kjh54ArE89qKZKNsCik5EZIEt","object":"chat.completion.chunk","created":1732513108,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[],"usage":{"prompt_tokens":53,"completion_tokens":5,"total_tokens":58,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}}
data: {"id":"chatcmpl-Al1DBOr3iDv5vEmEq6YxCSZ31FFeO","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[],"usage":{"prompt_tokens":53,"completion_tokens":5,"total_tokens":58,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}}


data: [DONE]
@@ -73,13 +73,13 @@ interactions:
CF-Cache-Status:
- DYNAMIC
CF-RAY:
- 8e7f4af0edbf238d-SJC
- 8fb5d726fd947ee7-LAX
Connection:
- keep-alive
Content-Type:
- text/event-stream; charset=utf-8
Date:
- Mon, 25 Nov 2024 05:38:28 GMT
- Wed, 01 Jan 2025 22:10:37 GMT
Server:
- cloudflare
Transfer-Encoding:
@@ -91,7 +91,7 @@ interactions:
alt-svc:
- h3=":443"; ma=86400
openai-processing-ms:
- '205'
- '226'
openai-version:
- '2020-10-01'
strict-transport-security:
@@ -109,19 +109,18 @@ interactions:
x-ratelimit-reset-tokens:
- 44ms
x-request-id:
- req_5df2a7cf8b9d982084c0c588bcce800d
- req_8879dc57c8e467a233a92438749a268e
status:
code: 200
message: OK
- request:
body: '{"messages": [{"role": "user", "content": "Return a country."}, {"content":
null, "refusal": null, "role": "assistant", "audio": null, "function_call":
null, "tool_calls": [{"id": "call_6Dk7MgfrU5OMIvVdcxiJ9TLE", "function": {"arguments":
"{\"name\":\"Australia\"}", "name": "return_country", "parsed_arguments": null},
null, "tool_calls": [{"id": "call_xnsDVVMJ4rh23jW9WEC22IyZ", "function": {"arguments":
"{\"name\":\"Canada\"}", "name": "return_country", "parsed_arguments": null},
"type": "function", "index": 0}], "parsed": null}, {"role": "tool", "tool_call_id":
"call_6Dk7MgfrU5OMIvVdcxiJ9TLE", "content": "1 validation error for Country\nname\n Value
error, Country must be Ireland. [type=value_error, input_value=''Australia'',
input_type=str]\n For further information visit https://errors.pydantic.dev/2.9/v/value_error"}],
"call_xnsDVVMJ4rh23jW9WEC22IyZ", "content": "1 validation error for Country\nname\n Value
error, Country must be Ireland. [type=value_error, input_value=''Canada'', input_type=str]"}],
"model": "gpt-4o", "parallel_tool_calls": false, "stream": true, "stream_options":
{"include_usage": true}, "tool_choice": {"type": "function", "function": {"name":
"return_country"}}, "tools": [{"type": "function", "function": {"name": "return_country",
@@ -135,7 +134,7 @@ interactions:
connection:
- keep-alive
content-length:
- '1046'
- '959'
content-type:
- application/json
host:
@@ -162,28 +161,28 @@ interactions:
uri: https://api.openai.com/v1/chat/completions
response:
body:
string: 'data: {"id":"chatcmpl-AXM5lf2YZmhdFaEv938fU8xSwn5ug","object":"chat.completion.chunk","created":1732513109,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"index":0,"id":"call_9492M1JOz1OVPrONfWjISFxN","type":"function","function":{"name":"return_country","arguments":""}}],"refusal":null},"logprobs":null,"finish_reason":null}],"usage":null}
string: 'data: {"id":"chatcmpl-Al1DBrEWRxUPR6zIyaXj9Prn1Zfk4","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"index":0,"id":"call_VE10M4nhW4rV1VX7Qi0qREeM","type":"function","function":{"name":"return_country","arguments":""}}],"refusal":null},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5lf2YZmhdFaEv938fU8xSwn5ug","object":"chat.completion.chunk","created":1732513109,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBrEWRxUPR6zIyaXj9Prn1Zfk4","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5lf2YZmhdFaEv938fU8xSwn5ug","object":"chat.completion.chunk","created":1732513109,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"name"}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBrEWRxUPR6zIyaXj9Prn1Zfk4","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"name"}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5lf2YZmhdFaEv938fU8xSwn5ug","object":"chat.completion.chunk","created":1732513109,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\""}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBrEWRxUPR6zIyaXj9Prn1Zfk4","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\""}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5lf2YZmhdFaEv938fU8xSwn5ug","object":"chat.completion.chunk","created":1732513109,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"Ireland"}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBrEWRxUPR6zIyaXj9Prn1Zfk4","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"Ireland"}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5lf2YZmhdFaEv938fU8xSwn5ug","object":"chat.completion.chunk","created":1732513109,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"}}]},"logprobs":null,"finish_reason":null}],"usage":null}
data: {"id":"chatcmpl-Al1DBrEWRxUPR6zIyaXj9Prn1Zfk4","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"}}]},"logprobs":null,"finish_reason":null}],"usage":null}


data: {"id":"chatcmpl-AXM5lf2YZmhdFaEv938fU8xSwn5ug","object":"chat.completion.chunk","created":1732513109,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null}
data: {"id":"chatcmpl-Al1DBrEWRxUPR6zIyaXj9Prn1Zfk4","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null}


data: {"id":"chatcmpl-AXM5lf2YZmhdFaEv938fU8xSwn5ug","object":"chat.completion.chunk","created":1732513109,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_7f6be3efb0","choices":[],"usage":{"prompt_tokens":125,"completion_tokens":5,"total_tokens":130,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}}
data: {"id":"chatcmpl-Al1DBrEWRxUPR6zIyaXj9Prn1Zfk4","object":"chat.completion.chunk","created":1735769437,"model":"gpt-4o-2024-08-06","system_fingerprint":"fp_5f20662549","choices":[],"usage":{"prompt_tokens":106,"completion_tokens":5,"total_tokens":111,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}}


data: [DONE]
@@ -194,13 +193,13 @@ interactions:
CF-Cache-Status:
- DYNAMIC
CF-RAY:
- 8e7f4af438da238d-SJC
- 8fb5d729a8327ee7-LAX
Connection:
- keep-alive
Content-Type:
- text/event-stream; charset=utf-8
Date:
- Mon, 25 Nov 2024 05:38:29 GMT
- Wed, 01 Jan 2025 22:10:37 GMT
Server:
- cloudflare
Transfer-Encoding:
@@ -212,7 +211,7 @@ interactions:
alt-svc:
- h3=":443"; ma=86400
openai-processing-ms:
- '276'
- '239'
openai-version:
- '2020-10-01'
strict-transport-security:
@@ -224,13 +223,13 @@ interactions:
x-ratelimit-remaining-requests:
- '499'
x-ratelimit-remaining-tokens:
- '29923'
- '29943'
x-ratelimit-reset-requests:
- 120ms
x-ratelimit-reset-tokens:
- 154ms
- 114ms
x-request-id:
- req_1c8ba63d5419f12f535bf10ab37d2ddc
- req_9f13b7185c0e0ee0be8a63fb28e5d0b3
status:
code: 200
message: OK
Loading