The loss of BatchFormer with MoCo-v3 ?

[iorange-77]

May I ask the 133-134 lines of code in ./moco-v3/moco/builder.py:
loss = self.contrastive_loss(q1[:N], k2[:N]) + self.contrastive_loss(q1[:N], k2[N:]) + self.contrastive_loss(q1[N:], k2[:N]) + self.contrastive_loss(q1[N:], k2[:N]) +
self.contrastive_loss(q2[:N], k1[:N]) + self.contrastive_loss(q2[:N], k1[N:]) + self.contrastive_loss(q2[:N], k1[:N]) + self.contrastive_loss(q2[:N], k1[N:])

Is the loss calculated incorrectly? should be: ？？
loss = self.contrastive_loss(q1[:N], k2[:N]) + self.contrastive_loss(q1[:N], k2[N:]) + self.contrastive_loss(q1[N:], k2[:N]) + self.contrastive_loss(q1[N:], k2[N:]) +
self.contrastive_loss(q2[:N], k1[:N]) + self.contrastive_loss(q2[:N], k1[N:]) + self.contrastive_loss(q2[N:], k1[:N]) + self.contrastive_loss(q2[N:], k1[N:])

If my understanding is wrong, what does this loss calculation method mean?

[zhihou7]

Dear @iorange-1014,
Many Thanks to you! It is a typo. It should be self.contrastive_loss(q2[N:], k1[:N]) + self.contrastive_loss(q2[N:], k1[N:]) in the last! I have updated the code.

Sincerely yours,

[zhihou7]

Hi @iorange-1014,
Thanks for your comment. Sorry for the terrible code. It seems like the model achieves clearly better performance with the revised code. The pretrained model is provided in here. Actually, I still think current losses might include redundant terms. I am not familiar with moco. And I do not have enough computational resources to tune this, but providing a naive and intuitive version. Lastly, many thanks to you.

Regards,