Fine-tuning on an uncolored labeled dataset

[accoumar12]

Hello,
Thanks for your amazing work!
I have a custom uncolored labeled dataset, with precise labels for the instances.
It seems that it does not really fit the SAM pipeline, where the SAM segmentation does not bring any value... Do you have any guideline for fine tuning ? I thought to totally change the pipeline and introduce directly the labels, modifying the loss for example.
Thanks for any tip!

[miscal]

I am not the author, but maybe you could use normal map for segmentation?

[yhyang-myron]

Hi, thanks for your interest in our work!
Perhaps you could directly use this data to train a 3D network by adopting SAM's training method and loss configuration?
Or conditioned on a 3D scale value?

[yhyang-myron]

For our design, we directly set the color value (after rgb augmentation) to (0,0,0) for uncolored data.