IOU thresholds during evaluation

[gholste]

I have two questions about default IOU thresholds used during evaluation that I don't see a way of tweaking without editing source code. Let me know if I'm misunderstanding something.

1. When running test.py, is there a fixed IOU threshold of 0.5 when determining true positives? In line 201 of test.py, I see for j in (ious > iouv[0]).nonzero(as_tuple=False): when identifying correct predictions, where iouv[0] is set to 0.5 earlier as per line 78. As far as I can tell, this means that only predicted boxes with an IOU > 0.5 with a GT box can be considered correct.
   
   I'm relatively new to object detection, but isn't this something one might want to change? E.g., if for my problem, I am okay with "overcalling" objects and increasing recall at the expense of precision, I might want to lower that threshold and obtain more true positives.

2. After "correct" boxes are identified, I see that ap_per_class is called to compute precision (P), recall (R), [email protected], and [email protected]:0.95. I only now realize that P and R are not returned according to --iou-thres (instead, this is for NMS), but actually come from the threshold with maximum F1 score. This being the case, how would I extract the P and R values for a desired threshold? Is it as simple as replacing line 77 of metrics.py with something like i = px.size // 2 to utilize a threshold around 0.5?

I'm aware I can change the source code however I like, but is there a reason these are fixed defaults that can't be tweaked through command-line arguments? Would you suggest against changing them? Any help would be appreciated, and again please let me know if I'm getting something wrong.

[glenn-jocher]

@gholste P and R values are computed over one thousand thresholds and displayed in your test results directory as figures. [email protected] naturally uses a 0.5 threshold by its definition, this is the entire point of the metric. Different metrics use different thresholds.

See https://en.wikipedia.org/wiki/Precision_and_recall

[gholste]

@glenn-jocher Thanks for the quick response! To your first point, I'm aware of this, but my question is how to actually extract the precision/recall value at a threshold (apart from eyeballing a figure).

I see what you're saying about the 0.5 threshold being used to calculate [email protected], but doesn't this choice also affect what we call a "correct" prediction and thus the elements of our confusion matrix and values of P and R? I guess I'm saying that for the purposes of metrics other than [email protected] and [email protected]:0.95, wouldn't it be conceivable to use a lower IOU threshold to calculate precision/recall/other confusion matrix-based metrics? And if so, I'm wondering how I would do that.