Is it possible to training with mixed data f segmentation and object detection?

[lhphanto]

Hi,

I have one small dataset which contains both bounding box and segmentation mask. And another larger dataset that has only bounding box ground truths. And my objective is to create a object detection model ( I am planning to use yolo-v7 but I feel like the main logic is very similar to yolo-v5 especially the segmentation part?).

Is it possible to utilize both datasets? my thoughts:
Idea 1. modify the segmentation dataloader, loss calculations - so that we use masks and do masks loss only if masks exist.
Idea 2. minimal changes to the code. Just train with segmentation data for about 10 epochs, save off checkpoint, then load checkpoint, switch to object detection model/data , train for 10 epochs, then switch back to segmentation data.

Seeking some helps and suggestions about:

1. Has anyone already did this somewhere I could learn from?
2. Any problems you see?

Thanks a lot!!

[parnia-ai]

Actually I have the same question. I'm encountering an issue where the objects of interest in my project are heavily occluded. To improve model performance, I need to use a combined dataset with both bounding boxes and segmentation masks, helping the model better identify these objects. However, the model discards segmentation data during training, which significantly takes a lot of my time!

Is there any other way to cope with this problem?
Thanks in advance