precision-recall functionality

[michaelbornholdt]

I am currently struggling to fully understand the functionality of the precision-recall operation.
Thus far I have taken a data sample from the LINCS database and followed the demo. Doing that I have noticed that I do not understand what the replicate_groups do.
I have done a first evaluation here

1. First I only set replicate_groups=["Metadata_moa"] but that only gave the precision column as output and all precisions were = 1.
2. Then I added "Metadata_broad_sample"] which resulted in a recall column - so I think I am using it correctly now. Not sure about the actual values tho.

I have spent a little time looking at the precision_recall.py but haven't gotten very far with understanding that.

So, again my question in short:
If Precision at k is the proportion of recommended items in the top-k set that are relevant, then why do we need two replicate_group values?

[gwaybio]

For operation="precision_recall", the variable replicate_groups is a list that notes which columns of your dataframe store replicate information. Setting replicate_groups=["Metadata_moa"] means that every profile with the same MOA is considered a replicate.

I do not know why you're observing the behavior you describe though! My only guess is that the return_bundle contains a missing value for recall and the pandas groupby operation that calculates precision_recall per group (here) cannot handle a missing value.

Ah! But maybe this is the fundamental issue... in that groupby command, with only one element listed in replicate_cols the code is only ever including replicates in the precision_recall calculation. In other words,

cytominer-eval/cytominer_eval/operations/util.py

Line 104 in 59684cf

recall_denom__total_relevant_items = sum(replicate_group_df.group_replicate)

is always the same number of rows as replicate_group_df.

[gwaybio]

I think this is bad, and we should try to fix.

The original goal was to stick with a consistent API for all metrics (evaluate.py), but maybe this is too restrictive

[michaelbornholdt]

So are we summing over one column there which has different replicate groups "in it"
I feel like you only want to be able to input one replicate group, not more - right?

[michaelbornholdt]

Ok, I have figured out why my first try did not create any sensible results. I was using the first for loop to query for a singe Moa and then if I calculate the precision, it will obviously be 1!

precision_recall_at_k = []

for moa in moa_labels:
    moa_df = df.query("Metadata_moa == @moa")

    for k in [2, 25]:

        precision_recall_results = evaluate(
            profiles=moa_df,
            features=features,
            meta_features=meta_features,
            replicate_groups=["Metadata_moa"],
            operation="precision_recall",
            similarity_metric="pearson",
            precision_recall_k=k
        ).assign(moa=moa)

        precision_recall_at_k.append(precision_recall_results)

So that code above is incorrect, right? I should be running precision-recall over the whole data frame?

[gwaybio]

Yes, try running over the whole data-frame, although I think my comment in #48 (comment) might still stand

[michaelbornholdt]

Running over the full df takes a lot longer! :D
Yes agreed, I think it shouldn't let in more than one rep group

[michaelbornholdt]

@gwaygenomics
Something is still fishy here, or I am not understanding it.
Can you have a look at this nb: https://github.com/broadinstitute/neural-profiling/blob/main/project_baseline/understanding%20recal.ipynb

Why are the precisions so low? shouldn't they be 100%? since we only have two MOAs

[michaelbornholdt]

This is the actual correlation matrix