Skip to content
This repository has been archived by the owner on May 17, 2024. It is now read-only.

Accept multiple evaluator in runner #173

Open
yujonglee opened this issue Oct 22, 2023 · 0 comments
Open

Accept multiple evaluator in runner #173

yujonglee opened this issue Oct 22, 2023 · 0 comments

Comments

@yujonglee
Copy link
Owner

e1 = fastrepl.RAGEvaluator(
    node=fastrepl.RAGAS(
        model="gpt-3.5-turbo",
        metric="Faithfulness",
    ),
)

e2 = fastrepl.RAGEvaluator(
    node=fastrepl.RAGAS(
        model="gpt-3.5-turbo",
        metric="AnswerRelevancy",
    ),
)

runner = fastrepl.local_runner(evaluators=[e1, e2], dataset=ds)
result = runner.run()

If we go this route, output_feature option should be evaluator level, not runner level. (Change needed)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant