Pytorch Lightning object detection performance worse than Rastervision pipeline with same configuration

[sethkelley2]

Hello,

I have created a PyTorch Lightning pipeline for doing object detection, much like the example, "Using Raster Vision with Lightning". I have tried to reproduce the results that I have achieved in object detection using raster-vision's pipeline, but I cannot seem to achieve the same performance. One observation that I have is that the loss is much lower in the rastervision pipeline from the first epoch compared to the loss that I experience after the first epoch in my Lightning Class.

I did my best to reproduce the exact configuration that I used in my raster-vision pipeline including the same:

• datasets (including the same uris, image sizes, ratios, batch sizes, etc..)
• same FasterRCNN model type with pretrained weights
• same loss function (the built in loss function to FasterRCNN torchvision class)
• same class configuration

I am unsure which optimizer that rastervision uses so I have used both Adam with the same learning rate and SGD with the same learning rate and weight decay = 0.0005.

My question is, are there any additional pre-processing steps that are happening in the rastervision pipeline that may be contributing to better performance? If not, might you have any inclination as to why I am seeing the poorer performance from the PyTorch Lightning instance?

I am a running Rastervision 0.20.2, Lightning 2.0.2, torch 1.12.1+cu116, torchvision 0.13.1+cu116

Thank you for your help! Regards,

Seth

[AdeelH]

Hi. Sorry for the late reply. It's not easy to say where the difference is without a deeper comparison, but I can try to answer your specific questions. How significant is the difference in performance?

I am unsure which optimizer that rastervision uses so I have used both Adam with the same learning rate and SGD with the same learning rate and weight decay = 0.0005.

Raster Vision uses an Adam optimizer with a one-cycle schedule. To get the default optimizer and scheduler that RV uses, you can create a SolverConfig, passing in your learning rate and batch size, and then call .build_optimizer() and .build_step_scheduler() on it.

My question is, are there any additional pre-processing steps that are happening in the rastervision pipeline that may be contributing to better performance?

RV also applies some simple data augmentations by default. These are: ['RandomRotate90', 'HorizontalFlip', 'VerticalFlip'].