Local LLM

[Gustavobrg]

Is it possible to use local models or are there any plans for that to happen? For example, using models from Hugging Face like the meta-llama/Llama-3.2-11B-Vision.

[torrischen]

also wondering

[chaoxxin]

also wondering

[cainmagi]

I have confirmed that currently the models on Huggingface cannot be used. See the following codes:

https://github.com/BerriAI/litellm/blob/7a5dd29fe05eb931e9c4ae743f4a862aabd3f488/litellm/utils.py#L4320-L4338

It can be found that if you are using a Huggingface model, supports_vision will be not set (default is None). However, zerox requires supports_vision to be True, see

zerox/py_zerox/pyzerox/models/modellitellm.py

Lines 63 to 66 in aa3d881

	def validate_model(self) -> None:
	'''Validates the model to ensure it is a vision model.'''
	if not litellm.supports_vision(model=self.model):
	raise NotAVisionModel(extra_info={"model": self.model})

Apparently, it means that the all Huggingface models will cause an error like this:

pyzerox.errors.exceptions.NotAVisionModel: 
    The provided model is not a vision model. Please provide a vision model

It seems that the only option to use the local model is to use ollama, which has been proved to be available, see

#106

---

Edited:

Sorry for mentioning that. I have not noticed that the results in #106 are totally non-sense. Now I can reproduce the same non-sense results. The model llama3.2-vision does not extract any meaningful information at all.

I decide to quit from this package. Apparently, this package is difficult to understand for somehow. I have succeeded using docling before. Maybe you can also try docling, which is not perfect but at least working.