Support Mistral's new visual model: Pixtral-12b-240910

[awaescher]

Mistral AI just dropped Pixtral, their 12b model with vision support.

• https://github.com/mistralai/mistral-common/releases/tag/v1.4.0
• https://www.reddit.com/r/LocalLLaMA/comments/1fe3x1z/mistral_dropping_a_new_magnet_link/

[Qualzz]

Gosh I love them.

[Kerimcanak]

Support.
I want to see this model to be usable on Ollama.

[tuanlda78202]

Hope too ~

[briansan]

+1

[awaescher]

Kudos to the Ollama team ❤️

(source)

[abenmrad]

Related to Pixtral, but more generally regarding Multimodal support in Ollama:

From my experiments today, Ollama is still supporting multi-modal chat with LLaVa (retried today with v0.3.10).

There were indeed some changes in the Llama.cpp server a while back.

I was genuinely interested to understand how Ollama can still handle it while Llama.cpp reportedly cannot anymore. Was Ollama relying on llama-cli, not llama-server?

Turns out it's not relying on either. Ollama is integrating directly with the llama.cpp code base. As for LLaVa support, they lifted the LLaVa support directly from the Llama.cpp server codebase and have been maintaining that in addition to everything else since then...

Ollama team are truly unsung heroes in this technological revolution.

[BennisonDevadoss]

Is it available at Ollama now?

[webboty]

Is it available at Ollama now?

ollama pull pixtral
pulling manifest
Error: pull model manifest: file does not exist

Not yet.

[dropout-developer]

Is it available at Ollama now?

ollama pull pixtral pulling manifest Error: pull model manifest: file does not exist

Not yet.

Same for me

[CRCODE22]

Tried to convert and add this with ollama https://huggingface.co/DewEfresh/pixtral-12b-8bit/tree/main but it seems the architecture is not supported by Ollama (yet).

K:\AI\DewEfresh\pixtral-12b-8bit>ollama create Pixtral-12B-2409
transferring model data 100%
converting model
Error: unsupported architecture

K:\AI\DewEfresh\pixtral-12b-8bit>ollama create --quantize q8_K_M Pixtral-12B-2409
transferring model data 100%
converting model
Error: unsupported architecture

[celobusana]

I tried to run the Pixtral with python code using an RTX 4060 with 16GB, but it was not possible :(. Perhaps it would work with a 4090 with 24GB.

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 60.00 MiB. GPU 0 has a total capacity of 15.70 GiB of which 34.69 MiB is free. Including non-PyTorch memory, this process has 15.64 GiB memory in use. Of the allocated memory 15.50 GiB is allocated by PyTorch, and 10.20 MiB is reserved by PyTorch but unallocated.

[thecr7guy2]

Cant wait to try the gguf version of Pixtral man
Ollama gonna deliver a christmas present early for us

[ddpasa]

I tried to run the Pixtral with python code using an RTX 4060 with 16GB, but it was not possible :(. Perhaps it would work with a 4090 with 24GB.

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 60.00 MiB. GPU 0 has a total capacity of 15.70 GiB of which 34.69 MiB is free. Including non-PyTorch memory, this process has 15.64 GiB memory in use. Of the allocated memory 15.50 GiB is allocated by PyTorch, and 10.20 MiB is reserved by PyTorch but unallocated.

You need to quantize to run an 12b model on 16GB hardware.

[Qualzz]

any news ?

[jasondalycan]

+1 for this feature

[A-REDA-EL]

any news ?

[frostworx]

please stop spamming here.

multiple people are subscribed to this issue and patiently wait until it is done.
all of them receive an email for every "+1" - it is annoying and for sure doesn't speed up anything.
thanks for your understanding.

(sorry for another mail, subscribers...)

[robert-mcdermott]

Anyone else checking the model library a couple times a day waiting for Pixtral, Llama3.2 9B and Molmo-7B to drop? 😄

[Qualzz]

im not sure if the team is actively working on multimodal support or if they're focusing on something else at the moment. What is certain is that multimodal capabilities will become increasingly essential in the near future, and many users may switch to alternatives that offer this functionality.

…

On Tue, Oct 1, 2024, 23:11 Robert McDermott ***@***.***> wrote: Anyone else checking the model library a couple times a day waiting for Pixtral, Llama3.2 9B and Molmo-7B to drop? 😄 — Reply to this email directly, view it on GitHub <#6748 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIMKMGFPFQGUUDYTKWQV67TZZMFWZAVCNFSM6AAAAABOANEDJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBXGA4DGMRUGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[sr-tream]

in the near future, and many users may switch to alternatives that offer this functionality.

Not with local models, I guess.

[Rakhsan]

How much long any hint pls?

[wijjj]

Anyone else checking the model library a couple times a day waiting for Pixtral, Llama3.2 9B and Molmo-7B to drop? 😄

pretty much

[nuryslyrt]

+1 to this feature :)

[Rakhsan]

Anyone else checking the model library a couple times a day waiting for Pixtral, Llama3.2 9B and Molmo-7B to drop? 😄

pretty much

there is no Llama3.2 9B THERE ARE 1, 3, 11, 90

[wijjj]

Anyone else checking the model library a couple times a day waiting for Pixtral, Llama3.2 9B and Molmo-7B to drop? 😄

pretty much

there is no Llama3.2 9B THERE ARE 1, 3, 11, 90

yeah. Thanks. You know what we mean. Anyways: waiting patiently.

[semidark]

please stop spamming here.

multiple people are subscribed to this issue and patiently wait until it is done.
all of them receive an email for every "+1" - it is annoying and for sure doesn't speed up anything.
thanks for your understanding.

(sorry for another mail, subscribers...)

Please do your Daily checks for "the drop" without spamming. Holy flipping cow.

[pbasov]

Just to point out an alternative for now - LM Studio just released with Pixtral support:
https://lmstudio.ai/blog/lmstudio-v0.3.5

[oderwat]

@pbasov Do I understand it right that this is Apple-Silicone only?

[pbasov]

@oderwat I believe so, yes, since it's enabled by the MLX engine and llama.cpp still doesn't support it.
ggerganov/llama.cpp#9440

But I'm sure ollama is going to get Pixtral support very soon, seeing that llama3.2 vision support is being rolled in 0.4

[Rakhsan]

guys I ma save you some time. I learned that pixtral will COME by end of december. top secret. now stop wasting your time checking every few days

[nonetrix]

@oderwat I believe so, yes, since it's enabled by the MLX engine and llama.cpp still doesn't support it. ggerganov/llama.cpp#9440

But I'm sure ollama is going to get Pixtral support very soon, seeing that llama3.2 vision support is being rolled in 0.4

Seems they are switching to their own inference engine as well, or at least for vision models. Honestly just wished they made some kind of patch set for llama cpp and maintained it, probably more productive imo if llama cpp doesn't want to add them currently

[Leeaandrob]

👍

[Tianoklein]

PLEASE!
Allow us to compare 2 or more images via ollama API!

[nonetrix]

PLEASE! Allow us to compare 2 or more images via ollama API!

I'm pretty sure this is a thing that the LLM itself has to support correct me if I'm wrong

[Leeaandrob]

Pixtral is avaialable for now? Best, +55 21 98244-8275 Em qui., 5 de dez. de 2024 às 09:12, fuggy ***@***.***> escreveu:

…

PLEASE! Allow us to compare 2 or more images via ollama API! I'm pretty sure this is a thing that the LLM itself has to support correct me if I'm wrong — Reply to this email directly, view it on GitHub <#6748 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABL5ZOG2G7ESRYQ7JUQVJMT2EA7JRAVCNFSM6AAAAABOANEDJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRQGE2DQOJRGE> . You are receiving this because you commented.Message ID: ***@***.***>

[frostworx]

oh my... I suggest delaying pixtral support by one week for every useless comment here...

(sorry again, subscribers)

unsubscribing