Nvidia Inter-node Bus Bandwidth for Ring AllReduce

[timlee14]

Hi, I have a question about the NVLink Switch in the Inter-node networking section of the network section network/README.md.

Say that I have a multi-node system with this set up:

• DGX A100 for each node
• 8 GPUs per node
• Intra-node connection: 6 NVSwitch 2.0 with 12 NVLink 3.0
• Inter-node connection: NVLink Switch, 8xConnectX-7, 200Gb/s InfiniBand

In implementing a ring algorithm with more than one node, my current understanding is that the rings are formed across nodes (from Nvidia GTC slide below).

Thus when performing an AllReduce, each step requires at least one inter-node communication somewhere in the ring. This means that the NIC bus bandwidth is the bottleneck. But this doesn't match this statement:

The links use the same speeds as NVLink - so when both are used inter- and intra-node bandwidth per link is the same.

How can inter- and intra-node bandwidth be the same if the NIC is operating at 200Gb/s?

Am I correct in assuming that the data has to go through the 200Gb/s NIC? Does using NVLink Switch somehow bypass the NIC? If so, how is the data moved between nodes?

[stas00]

Thank you for this interesting question, @timlee14

I don't have access to a system that has NVSwitch w/ NVLink inter-node to validate

But that is what it says here https://www.nvidia.com/en-us/data-center/nvlink/ and quite a few other nvidia docs - where it suggests that it can interconnect more than 8 gpus at the NVLink speed, which creates a sort of super-node with up to 256 GPUs interlinked at NVLink speed.

I suppose that NVSwitch can also work with IB instead of NVLink, in which case it'd be a slower inter-node and as you correctly said would be the slowest link in the chain.

What do you think?

I'd have been much better for me to find the truth if I had the access to such a system, but w/o the access I have to rely on the documentation.