The NVLink Leash

Tensor parallelism is almost always kept within a single node's NVLink domain, while pipeline and data parallelism routinely span nodes over InfiniBand. What is the real reason for this split?

InfiniBand's protocol only supports data parallelism traffic patterns, not pipeline stage transfers

Tensor parallelism needs an all-reduce of full activation tensors after nearly every layer, and InfiniBand's inter-node bandwidth (tens of GB/s per link) is far below NVLink's intra-node bandwidth (hundreds of GB/s to ~1 TB/s), so spreading it across nodes bottlenecks on the network

Pipeline and data parallelism perform no network communication between devices, so bandwidth doesn't matter for them

NVLink hardware can only perform point-to-point transfers and has no support for collective operations like all-reduce