ENH: Add a new metric to summarize the best possible values / thresholds for a given metric and label

[ntlind]

Feature Type

• Adding new functionality to valor

• Changing existing functionality in valor

• Removing existing functionality in valor

Feature Description

Initial context can be found in this discussion.

Depending on their use case, users may want to see:
- All metrics and thresholds for all predictions, even those which didn't have the highest score on a given groundtruth (this is what we show today in DetailedPrecisionRecallCurve)
- All metrics and thresholds for only those predictions which had the highest confidence score for a given groundtruth (users can get some of this from our standard metrics, but not at the same level of granularity as exists for the bullet above)
- A summary of the best possible score that can be achieved for a given label, and how achieving that score impacts other metrics

This ticket focuses on #3, where we want to users to understand what the best possible score would be for each metric / label. For example, the user would be able to answer "Given I want to optimize the F1 score for a particular label, give me the score and confidence threshold that would allow me to do that. Also, tell me how this threshold would impact the other labels and give me a few examples for TP, FP, and FN".

This functionality could be encapsulated in a separate metric as shown:

# TODO clarify if we'd need to add examples here?
{
    "F1": { # if you wanted to optimize F1 score....
        "(class, dog": { # for this particular label....
            "best_threshold": .45 # the maximum threshold that would optimize for this metric and class
            "values": { # the actual values if you were to choose this threshold
                "F1": .88,
                "precision": .66,
                ...
            },
        },
    },
}