Fixes in LLMJudge

[lilacheden]

1. allow changing main_score and score_prefix (and change default from "score" which doesn't support score_prefix)
2. Support embedded task data fields in LLMJudge template

[coveralls]

coverage: 79.384% (-0.04%) from 79.426%
when pulling 9754a63 on fix_llmjudge
into 0ed5ff6 on main.

[OfirArviv]

why the change?

[lilacheden]

to be able to add nested fields. For examples, a user asked to use only the instruction from the original template without the full source, this way we can send metadata/template/instruction

[OfirArviv]

@elronbandel is it an acceptable way to do it, that is fully supported, and won't be prone to changes in the future?

[OfirArviv]

anyway you need to add a documentation to this @lilacheden

[lilacheden]

@OfirArviv @elronbandel -
If you want it even cleaner we can support a dict of name:possibly-nested field as the context_fields

[yoavkatz]

Yes. I think @martinscooper highlighted the need to be able to rename context fields.

[lilacheden]

ok, added it

[martinscooper]

Does this allow for renaming context field then? If so, do the keys correspond to the final context names used in the prompts and the values correspond to the task_data key names?

I think this is useful to adapt the task data to the criteria. For example, the squad dataset uses the term context but the coherence criteria description uses original text

[lilacheden]

@martinscooper - yes, this way you can send a dictionary where the key is the name of the field in the prompt and the value is the field name (or path) in the task data.
e.g. {"instructions":"metadata/template/instruction"} - the prompt will mention context instructions

If a list is sent then the behavior is as before - each item is both the key and the value:
["question"] -> {"question":"question"}

[martinscooper]

FYI: @elronbandel, @yoavkatz and I have been discussing and working on improving the reported scores for both direct and pairwise evaluators. The changes are included in this PR. You can look at this and this commits for direct score changes, and this other one for pairwise score changes.

In summary:

• direct evaluators's main_score will be the criteria name (iff available and the criteria is the same for all instances), e.g. faithfulness. If not, a generic main_score llm_as_judge is used. A more granular score is included too, to avoid name conflicts if multiple metrics are used, it is {main_score}_{model_name}_{provider}.
• pairwise evaluator's main_score is the first system's winrate -called 1_winrate-. All other system's winrate and its mean value are reported too.

[yoavkatz]

FYI: @elronbandel, @yoavkatz and I have been discussing and working on improving the reported scores for both direct and pairwise evaluators. The changes are included in this PR. You can look at this and this commits for direct score changes, and this other one for pairwise score changes.

In summary:

• direct evaluators's main_score will be the criteria name (iff available and the criteria is the same for all instances), e.g. faithfulness. If not, a generic main_score llm_as_judge is used. A more granular score is included too, to avoid name conflicts if multiple metrics are used, it is {main_score}_{model_name}_{provider}.
• pairwise evaluator's main_score is the first system's winrate -called 1_winrate-. All other system's winrate and its mean value are reported too.

Yes, I think we should not change the score names in this PR, and wait for #1467 for the changes.

[lilacheden]

FYI: @elronbandel, @yoavkatz and I have been discussing and working on improving the reported scores for both direct and pairwise evaluators. The changes are included in this PR. You can look at this and this commits for direct score changes, and this other one for pairwise score changes.
In summary:

• direct evaluators's main_score will be the criteria name (iff available and the criteria is the same for all instances), e.g. faithfulness. If not, a generic main_score llm_as_judge is used. A more granular score is included too, to avoid name conflicts if multiple metrics are used, it is {main_score}_{model_name}_{provider}.
• pairwise evaluator's main_score is the first system's winrate -called 1_winrate-. All other system's winrate and its mean value are reported too.

Yes, I think we should not change the score names in this PR, and wait for #1467 for the changes.

@yoavkatz - reverted this main score change, can you approve the remaining change?

[martinscooper]

Could you remove this change?

[lilacheden]

done