Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#1008: Working <|image|> injection #1015

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

bioshazard
Copy link

I had trouble getting the latest master to work, so I branched off b38c72c since that was the last published Docker image. I was able to get image payloads working without needing to include <|image|> in the text part. I seem to run out of CUDA memory if I try a second image, but this will work for my single-image payload needs for now. I suspect I am wasting memory by duplicating the content per message, so some pointers on optimal rust would be appreciated or feel free to tweak this as needed.

I know mllama expects the <|image|> token in there. If any backends DONT expect the token in there, you might need to move this into the mllama backend itself to take care of its own need for such a token. Either way, I will be using this branch on my fork for myself, but I hope you can find it useful for #1008 if you the use case and approach is compelling. Thanks!

Copy link

Code Metrics Report
  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           35           28            0            7
 Dockerfile              1           41           22           10            9
 JSON                   12          105          104            0            1
 Python                 63         2706         2338           71          297
 Shell                   1           57           22           18           17
 Plain Text              3         3723            0         2413         1310
 TOML                   18          605          539            2           64
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       4            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               43         3333            0         2526          807
 |- BASH                 6          103          100            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                12          406          344            0           62
 |- TOML                 2           75           63            0           12
 (Total)                           4050          628         2526          896
-------------------------------------------------------------------------------
 Rust                  296        89600        80403         1861         7336
 |- Markdown           143         1593           25         1448          120
 (Total)                          91193        80428         3309         7456
===============================================================================
 Total                 445       100226        83475         6903         9848
===============================================================================
  

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant