[QUESTION] Megatron-LM DistributedOptimizer or NeMo MegatronDistributedFusedAdam Optimizer?

[TJ-Solergibert]

Hi,

After going across both Megatron-LM & NeMo I've found that NeMo configs set by default the MegatronDistributedFusedAdam optimizer from the NeMo framework. But Megatron also contains a DistributedOptimizer.

The NeMo one is based on the Apex's DistributedFusedAdam which incorporates ZeRO-2. I would like to know which one is better, both in terms of throughput and performance or which config do you recommend to use for large scale training.

Thanks!