How to stop model from loading every time?

[nivibilla]

Hey , sorry if this is a stupid question, but when using the high level api it takes 8sec to load the model every time I do inference. Is there a way to load the model once and do inference?

Trying to do inference in a loop

thanks

[BetaDoggo]

That sounds like an issue with your code. Make sure that your are not accidentally looping the Llama(model_path="model-path") line.
Here's a basic example I made where the model is only loaded once at the start then inference is run 5 times:

llm = Llama(model_path="model-path")
i=0
while (i < 5):
	output = llm("The sky is", max_tokens=20, echo=True)
	print(output)
	i = i + 1

[nivibilla]

It's so strange, I ran this code and my previous code and it seems to work fine.
But when I ran it yesterday night it seems to spend 8sec loading every time. I didn't change the code at all.

[nivibilla]

Thanks anyway though!