You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Using chain (ConversationalChainBuilder or Agent) does not support images.
prompt_args!{"input" => input,}
does not allow to pass an image url.
If the url is passed as a string the in input, an error is thrown OpenAI error: stream failed: Invalid status code: 400 Bad Request probably because the image is not under the right structure (Image content part).
Also langchain_rust::schemas::Message::messages_to_string only serialize text content. It does not take in consideration images.
let chain = ConversationalChainBuilder::new().llm(llm).build();let base64_image_url = "data:image/jpeg;base64,23434234234234".to_string();let input = format!("What is that? {}", base64_image_url);let result = chain.invoke(prompt_args!{"input" => input,});
Expected behavior
A way to pass image to a chain or at minimum an Agent.
Since images are on langchain_rust::schemas::Message structure, I would expect them to work in every aspect of the library.
Additional context
This is some kind of a violation of the Liskov Principle & Interface Segregation Principles. If image are available on langchain_rust::schemas::Message they should be supported everywhere in the library.
The text was updated successfully, but these errors were encountered:
Describe the bug
Using chain (
ConversationalChainBuilder
orAgent
) does not support images.does not allow to pass an
image url
.If the
url
is passed as a string the in input, an error is thrownOpenAI error: stream failed: Invalid status code: 400 Bad Request
probably because the image is not under the right structure (Image content part
).Also
langchain_rust::schemas::Message::messages_to_string
only serialize text content. It does not take in consideration images.langchain-rust/src/schemas/messages.rs
Lines 149 to 155 in 891c56f
To Reproduce
Expected behavior
A way to pass image to a chain or at minimum an Agent.
Since images are on
langchain_rust::schemas::Message
structure, I would expect them to work in every aspect of the library.Additional context
This is some kind of a violation of the
Liskov Principle
&Interface Segregation Principles
. If image are available onlangchain_rust::schemas::Message
they should be supported everywhere in the library.The text was updated successfully, but these errors were encountered: