Verifying AToMiC regressions

[lintool]

At Pyserini commit e68d54 (2024/10/12), I'm running:

nohup python -m pyserini.2cr.atomic --all --directory runs/ --display-commands >& logs/log.atomic &

Getting an error:

condition base-t2i:
  - Model: ViT-L-14.laion2b_s32b_b82k

      MRR@10 : 0.0685 [OK]
      R@10   : 0.1171 [OK]
      R@1000 : 0.4450 [FAIL] expected 0.4597

Final output:

Metric = MRR@10
                                   small-t2i     small-i2t     base-t2i     base-i2t     large-t2i     large-i2t     
ViT-L-14.laion2b_s32b_b82k         0.314         0.316         0.069        0.072        0.054         0.039         
ViT-H-14.laion2b_s32b_b79k         0.000         0.000         0.000        0.000        0.062         0.051         
ViT-bigG-14.laion2b_s39b_b160k     0.000         0.000         0.000        0.000        0.074         0.061         
ViT-B-32.laion2b_e16               0.000         0.000         0.000        0.000        0.031         0.025         
ViT-B-32.laion400m_e32             0.000         0.000         0.000        0.000        0.025         0.018         
openai.clip-vit-base-patch32       0.000         0.000         0.000        0.000        0.022         0.013         
openai.clip-vit-large-patch14      0.000         0.000         0.000        0.000        0.058         0.026         
Salesforce.blip-itm-base-coco      0.000         0.000         0.000        0.000        0.007         0.007         
Salesforce.blip-itm-large-coco     0.000         0.000         0.000        0.000        0.011         0.009         
facebook.flava-full                0.000         0.000         0.000        0.000        0.003         0.004         

Metric = R@10
                                   small-t2i     small-i2t     base-t2i     base-i2t     large-t2i     large-i2t     
ViT-L-14.laion2b_s32b_b82k         0.490         0.480         0.117        0.120        0.092         0.076         
ViT-H-14.laion2b_s32b_b79k         0.000         0.000         0.000        0.000        0.108         0.098         
ViT-bigG-14.laion2b_s39b_b160k     0.000         0.000         0.000        0.000        0.128         0.117         
ViT-B-32.laion2b_e16               0.000         0.000         0.000        0.000        0.052         0.047         
ViT-B-32.laion400m_e32             0.000         0.000         0.000        0.000        0.037         0.034         
openai.clip-vit-base-patch32       0.000         0.000         0.000        0.000        0.042         0.026         
openai.clip-vit-large-patch14      0.000         0.000         0.000        0.000        0.102         0.052         
Salesforce.blip-itm-base-coco      0.000         0.000         0.000        0.000        0.014         0.013         
Salesforce.blip-itm-large-coco     0.000         0.000         0.000        0.000        0.020         0.018         
facebook.flava-full                0.000         0.000         0.000        0.000        0.008         0.008         

Metric = R@1000
                                   small-t2i     small-i2t     base-t2i     base-i2t     large-t2i     large-i2t     
ViT-L-14.laion2b_s32b_b82k         0.945         0.942         0.445        0.460        0.360         0.334         
ViT-H-14.laion2b_s32b_b79k         0.000         0.000         0.000        0.000        0.405         0.388         
ViT-bigG-14.laion2b_s39b_b160k     0.000         0.000         0.000        0.000        0.443         0.433         
ViT-B-32.laion2b_e16               0.000         0.000         0.000        0.000        0.243         0.234         
ViT-B-32.laion400m_e32             0.000         0.000         0.000        0.000        0.184         0.175         
openai.clip-vit-base-patch32       0.000         0.000         0.000        0.000        0.244         0.187         
openai.clip-vit-large-patch14      0.000         0.000         0.000        0.000        0.407         0.273         
Salesforce.blip-itm-base-coco      0.000         0.000         0.000        0.000        0.109         0.110         
Salesforce.blip-itm-large-coco     0.000         0.000         0.000        0.000        0.149         0.137         
facebook.flava-full                0.000         0.000         0.000        0.000        0.143         0.129       

hey @justram Wanted to run this output by you...

• What's with the FAIL?
• Why so many 0.000's in the final output?

[justram]

OK, let me check what happened.

[lintool]

I've merged in #2014 - the merged in code produces exactly the same results as above, so whatever happened isn't due to that PR.