Question about FeatUp

[zcc00210]

Hello,
Thanks for your amazing work! I have a few questions about using Featup:

1. How do you use FeatUp to enhance the pixelated features of DINOv2? Is it for a vector n * H * W (n is the dimension of features) or an image? If it is convenient, could you please mark the relevant code?
2. When FeatUp is used, is ckpt without hierarchical specifications being used? Are there any differences between the two ckpt types in enhancing the features of DINO v2?
   Looking forward to your reply!

[yhyang-myron]

Hi, thanks for your interest in our work!

1. We use FeatUp in our pre-training stage. We distill the feature from FeatUp-DINOv2 to the 3D backbone via a MSE Loss.
2. What does a ckpt without hierarchical specifications refer to?

[zcc00210]

Hi, thanks for your interest in our work!

1. We use FeatUp in our pre-training stage. We distill the feature from FeatUp-DINOv2 to the 3D backbone via a MSE Loss.
2. What does a ckpt without hierarchical specifications refer to?

Sorry for my wrong expression of the second question, I want to ask, do you use checkpoint with No LayerNorm (as shown below)?

May I ask that does the pre-training stage refer to the process of building PTv3-object. And the corresponding code is not released?
As I want to use FeatUp in the pixel-by-pixel feature map at reduced resolution, but I don't know much about its network structure. So I want to learn from your work and see how FeatUp applies to feature vectors. If you can provide the relevant code, I would be grateful!!

[yhyang-myron]

Hi, the FeatUp we used is with additional LayerNorm:
torch.hub.load("mhamilton723/FeatUp", 'dinov2')

And the code is like this:

self.Encoder_2d = torch.hub.load("mhamilton723/FeatUp", 'dinov2').cuda().eval()
img_feat = self.Encoder_2d(imgs)
img_feat = F.interpolate(img_feat, size=(512, 512))
img_feat = rearrange(img_feat, '(b s) c h w -> b s c h w', b=B)
loss = F.mse_loss(point_feat, img_feat)