Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes in LLMJudge #1498

Merged
merged 7 commits into from
Jan 16, 2025
Merged

Fixes in LLMJudge #1498

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c94bfc5
bugfix in LLMJudge - allow changing main_score and score_prefix
lilacheden Jan 12, 2025
b0b9dfc
Support embedded task data fields in LLMJudge template
lilacheden Jan 12, 2025
9754a63
add main_score to results
lilacheden Jan 12, 2025
42e84f6
support dict llmjudge contexts_fields
lilacheden Jan 12, 2025
5c9ddc8
Revert "bugfix in LLMJudge - allow changing main_score and score_prefix"
lilacheden Jan 13, 2025
6cb0c3c
Revert "add main_score to results"
lilacheden Jan 13, 2025
3dcf444
Merge branch 'main' into fix_llmjudge
elronbandel Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions src/unitxt/llm_as_judge.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

from .api import infer
from .artifact import fetch_artifact
from .dict_utils import dict_get
from .error_utils import UnitxtError
from .inference import (
InferenceEngine,
Expand Down Expand Up @@ -59,7 +60,7 @@ class LLMJudge(BulkInstanceMetric):
# )
evaluator_name: EvaluatorNameEnum = None
check_positional_bias: bool = True
context_fields: str = ["context"]
context_fields: Union[str, List[str], Dict[str, str]] = ["context"]
generate_summaries: bool = True
format = "formats.chat_api"
include_prompts_in_result: bool = False
Expand All @@ -71,6 +72,10 @@ def prepare(self):
super().prepare()
if isinstance(self.context_fields, str):
self.context_fields = [self.context_fields]
if isinstance(self.context_fields, List):
self.context_fields = {
context_field: context_field for context_field in self.context_fields
}

# if not isinstance(self.option_selection_strategy, OptionSelectionStrategyEnum):
# self.option_selection_strategy = OptionSelectionStrategyEnum[
Expand Down Expand Up @@ -149,8 +154,8 @@ def get_contexts(self, task_data: List[Dict[str, Any]]) -> List[Dict[str, str]]:
return [
get_parsed_context(
{
context_field: td[context_field]
for context_field in self.context_fields
context_field_name: dict_get(td, context_field)
for context_field_name, context_field in self.context_fields.items()
}
)
for td in task_data
Expand Down
Loading