Ring's Constant Toll

In a ring all-reduce across N GPUs on a tensor of total size K bytes, each GPU sends and receives data over 2(N-1) pipelined steps of K/N bytes each. As N grows large, what does the total data moved through each GPU's link approach?

2K/N, shrinking to zero as N grows

K/2, a constant independent of both K and N

2K, independent of N

2K log2(N), growing with N

K*N, growing linearly with N