[QUESTION]Why is the operator of computation slower when computation overlaps with communication

[yu-depend]

Why is the operator of computation slower when computation overlaps with communication

[elliottnv]

This question is too general. Can you provide more details?

[yu-depend]

When -- overlap-grad-Reduce is used, the compute kernel with reduce_scatter overlap (the duration of GEMM kernel with red rectangle in the figure is longer than that of kernel without overlap). I assume that the NCCL kernel seized the SM resource?Please explain it.

[deepakn94]

Right, there is some resource contention. This is expected.

[yu-depend]

Thank you for your reply.