Using llama3.2 as local llm for deepeval

[lorenzofilitti]

Hi I wanted to test deepeval. I set llama3.2:latest as my local model with the command: deepeval set-local-model --model-name=llama3.2:latest --base-url="http://localhost:11434/v1/" --api-key="ollama".

When I run the example code on deep eval website and I specify the model I want to use(llama3.2:latest) it doesn't find it. How can I solve this? The code:

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

correctness_metric = GEval(
    name="Correctness",
    criteria="Determine whether the actual output is factually correct based on the expected output.",
    # NOTE: you can only provide either criteria or evaluation_steps, and not both
    evaluation_steps=[
        "Check whether the facts in 'actual output' contradicts any facts in 'expected output'",
        "You should also heavily penalize omission of detail",
        "Vague language, or contradicting OPINIONS, are OK"
    ],
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT, 
    LLMTestCaseParams.EXPECTED_OUTPUT],
    model=llama3.2:latest
)

[antoninoLorenzo]

That's not clear how you are running the code, there are two approaches:

• using deepeval command
• manually running the code

I think that when you set a local LLM like this you should use the first approach.

Personally I suggest using the second approach for higher flexibility, however this involves writing a DeepEvalBaseLLM subclass that you pass as an instance to the model parameter; refer to Using Custom LLMs for Evaluation.

[soyboy17]

Hey Lorenzo! I found I needed to set it up in a similar way to Google VertexAI on the Metrics Introduction page (with a few adjustments). Apologies for the formatting, but this is what worked for me

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams, LLMTestCase

Modules for model implementation

from deepeval.models.base_model import DeepEvalBaseLLM
from langchain_ollama import ChatOllama

Create a class to implement the local Ollama Llama model for DeepEval

class OllamaLlama(DeepEvalBaseLLM):
def init(self, model):
self.model = model

def load_model(self):
    return self.model

def generate(self, prompt: str) -> str:
    chat_model = self.load_model()
    return chat_model.invoke(prompt).content

async def a_generate(self, prompt: str) -> str:
    chat_model = self.load_model()
    res = await chat_model.ainvoke(prompt)
    return res.content

def get_model_name(self):
    return "llama3.1:latest"

Config for the local Llama model

custom_model_llama = ChatOllama(
model="llama3.1:latest",
base_url="http://localhost:11434/",
api_key="ollama"
)

Instantiate the model

llama_model = OllamaLlama(custom_model_llama)

Define your test case

test_case = LLMTestCase(
input = "What if these shoes don't fit?",
actual_output = "We offer a 30-day full refund at no extra cost.",
expected_output = "You're eligible for a free full refund within 30 days of purchase.",
)

correctness_metric = GEval(
name="Correctness",
#criteria="Determine whether the actual output is factually correct based on the expected output.",
# NOTE: you can only provide either criteria or evaluation_steps, and not both
evaluation_steps=[
"Check whether the facts in 'actual output' contradicts any facts in 'expected output'",
"You should also heavily penalize omission of detail",
"Vague language, or contradicting OPINIONS, are OK"
],
evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT,
LLMTestCaseParams.EXPECTED_OUTPUT],
model=llama_model
)

Print completeness score and reason

correctness_metric.measure(test_case)
print("Correctness: ", correctness_metric.score)
print("Reason: ", correctness_metric.reason)

[antoninoLorenzo]

Yes, that's what I was talking about (except the formatting 😭).

As I said this approach is more flexible, for example I am working without a framework (like langchain) and with Ollama deployed on Google Colab; their support for custom LLMs made it possible for me to use deepeval as evaluation framework and in general it seems like the project owners actually care about usability, even for specific use cases.