[Question] Combining sync batchnorm with DeepSpeed?

[Parskatt]

Hi,

I'm not sure how to combine synchronized batchnormalization with deepspeed.
Using model_engine.module = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model_engine.module)
Gives me the error: AttributeError: SyncBatchNorm is only supported within torch.nn.parallel.DistributedDataParallel

Is there another way of making my batchnorms synchronized?

[Parskatt]

I solved it (I think) by just requiring dist_init_required=True in the initialization step.

[Parskatt]

Nevermind, that was simply because I was using a single GPU, its still not working for multiple GPUs for me.

[Parskatt]

Not sure what I was doing when I typed this, but nowadays it works by doing

model = nn.SyncBatchNorm.convert_sync_batchnorm(model)
model_engine, _, _, _ = deepspeed.initialize(model = model, ...)