I've also modified the tensorflow shift function. Unfortunately we need this function twice, once in numpy and once in tensorflow ops. The evaluation on COCO, using the same weights as before, give the following results:

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.286
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.530
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.280
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.170
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.336
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.368
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.257
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.424
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.464
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.315
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.523
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.567

The original results:

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.350
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.537
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.374
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.191
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.383
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.472
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.306
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.491
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.533
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.345
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.577
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.681

There's a difference there, which makes sense. The difference for IoU=0.5 is rather small, but the difference for IoU=0.75 is large (causing the difference in mAP to be relatively large as well). Basically all detections have been shifted a few pixels to the top left corner, which isn't that critical with a lenient overlap threshold of 0.5, but for 0.75 it is more severe.

Next I'll start a training process to finetune the network for this new setting, and add unit tests for this function.

Added unit tests and started training starting from the previous best weights. The first epoch moved the mAP from 0.286 to 0.333, so that's a nice first increase. I'll see how far it gets, hopefully it becomes at least as accurate as the original model :)

I'm awaiting the results of training (and that of keras-maskrcnn) before merging this. We'll have to tag it as a new version as well.

This is still on my radar btw, but it takes a while to retrain. So far the best mAP I got was 0.343, I'm trying to see if I can push it closer to 0.350 (with simply more training).

Thanks for the update, Hans!

It should be the same, right? For models that use ResNet50 as backbone the new computation gives the same result as the previous computation. The difference would occur for models that play with the padding or strides differently.

Thanks for the update, Hans!

It should be the same, right? For models that use ResNet50 as backbone the new computation gives the same result as the previous computation. The difference would occur for models that play with the padding or strides differently.

Yeah it should be, but the 0.350 mAP score from before was basically luck. I never succeeded in reproducing that exact score ;) a score of 0.34 is fine, but there was only one snapshot that reached 0.350.

Is there a reason why you are not using seeds for training?

Very good question: no there's not :D

Too late now ;)

😢 Would it be worth creating a PR which allows users to set seeds using a parameter and by default they aren't set?

Yeah, I would think it can be useful, definitely.

I've started training from scratch, but after a while the training stopped improving (achieved around 0.27 mAP). I've restarted training multiple times (starting from the weights of the previous iteration) and have pushed the mAP to 0.292 so far.

This annoys me a bit because it suggests that the training procedure is not working properly (which I've known for a while now). Perhaps a different learning rate scheduler, or different optimizer, or both? I don't have time to look into these settings, so if someone wants to help then it is much appreciated.

Until then, I'll keep restarting training after it stagnated and see how far we get :)

For the record, this PR is still valid, but I haven't got the time to properly train a new model that gets similar accuracy as the current model.

Have a question about this PR. Could this PR potentially improve models accuracy after we additionally train them with correct data?