Dividing contrastive loss by number of replicas -- correct?

[piotrmwojcik]

I think that in case when we are using only contrastive loss, dividing it by the number of replicas isn't doing what it is supposed to do. For every replica, the numerator consists of positive pairs for one replica-batch and the denominator contains every negative pair from the enlarged batch.