Can queries really be permuted independently?

[danr]

Under the header Assigning significance to mAP scores you write

To generate mAP distribution under the null hypothesis, we repeatedly reshuffle the rank list and recalculate mAP.

You refer to the rank list. But there is one rank list for each query profile. Looking at your code it seems you treat these rank lists independently. But would it not be more correct to shuffle the profile labels (query or reference), then calculate mAP by recalculating each query profile's rank list? Why is your shortcut correct?

Thanks!