4 / 10 Parallelism & Collectives

Ring's Constant Toll

In a ring all-reduce across N GPUs on a tensor of total size K bytes, each GPU sends and receives data over 2(N-1) pipelined steps of K/N bytes each. As N grows large, what does the total data moved through each GPU's link approach?

A

2K/N, shrinking to zero as N grows

B

K/2, a constant independent of both K and N

C

2K, independent of N

D

2K log2(N), growing with N

E

K*N, growing linearly with N