Add support for Phi-3.5-vision-instruct

[abetlen]

Adds support for Phi-3.5-vison-instruct

Phi-3.5-vision is a lightweight, state-of-the-art open multimodal model built upon datasets which include - synthetic data and filtered publicly available websites - with a focus on very high-quality, reasoning dense data both on text and vision.

The biggest challenge here was porting over the hd_transform that's performed on the image embeddings before the projector. This requires doing a 6d reshape-permute which I had to adapt to work with a series of 4d reshape-permutes. It looks like a mess but it does work (I tested this section against numpy).

Converted GGUFs can be found here

Prompt format

<|user|>\n<|image_1|>\n{prompt}<|end|>\n<|assistant|>\n

Update
Just realised this still needs work to implement the hd transform in the preprocessing steps as well, will keep it as a draft for now.

[ThiloteE]

Partially resolves #9119

[ayttop]

wher Phi-3.5-MoE-instruct gguf?

[Milor123]

@abetlen , Bro I've got this bug

Could please explain me how should use it in ollama Modelfile:

<|user|>\n<|image_1|>\n{prompt}<|end|>\n<|assistant|>\n

I've created the it using

my Modelfile:

PARAMETER num_ctx 12000
FROM Phi-3.5-3.8B-vision-instruct-Q8_0.gguf
FROM Phi-3.5-3.8B-vision-instruct-mmproj-F16.gguf
TEMPLATE "<|user|>\n<|image_1|>\n{prompt}<|end|>\n<|assistant|>\n"
PARAMETER stop <INST>
PARAMETER stop </INST>
PARAMETER stop <|end|>

but when run happened it

ollama run Phi-3.5-3.8B-vision-instruct-Q8_0             
Error: llama runner process has terminated: GGML_ASSERT(ggml_can_mul_mat(a, b)) failed

What should I change? I am running ollama from docker