#1008: Working <|image|> injection #1015
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I had trouble getting the latest
master
to work, so I branched offb38c72c
since that was the last published Docker image. I was able to get image payloads working without needing to include<|image|>
in the text part. I seem to run out of CUDA memory if I try a second image, but this will work for my single-image payload needs for now. I suspect I am wasting memory by duplicating the content per message, so some pointers on optimal rust would be appreciated or feel free to tweak this as needed.I know mllama expects the
<|image|>
token in there. If any backends DONT expect the token in there, you might need to move this into the mllama backend itself to take care of its own need for such a token. Either way, I will be using this branch on my fork for myself, but I hope you can find it useful for #1008 if you the use case and approach is compelling. Thanks!