The 1.5x Tax

Standard multi-GPU data parallelism synchronizes gradients with a ring all-reduce, decomposed into a reduce-scatter phase (moves ~Ψ elements per GPU) followed by an all-gather phase (moves another ~Ψ elements per GPU), for a baseline of ~2Ψ total traffic per parameter per step. ZeRO-3 / FSDP additionally shards the parameters themselves across ranks (not just optimizer states and gradients). What is ZeRO-3's per-parameter communication volume per training step, and why does it land at exactly 1.5x the baseline DP volume rather than 1x or 2x?

Ψ, since only the gradients need to be reduce-scattered; the parameters themselves never need to be gathered because each rank computes only on its own shard of every layer.

Ψ/N (N = world size), since sharding the parameters divides the total communicated volume by the sharding degree.

3Ψ: parameters must be reconstituted via all-gather once before the forward pass and again before the backward pass (since each rank only holds a shard at rest), then gradients are reduce-scattered once in the backward pass, so Ψ + Ψ + Ψ = 3Ψ, i.e. 1.5x the 2Ψ baseline.

2Ψ, identical to baseline DP, because sharding parameters, gradients, and optimizer states adds no extra collectives beyond the standard gradient all-reduce.

6Ψ, because the reduce-scatter and all-gather collectives are each internally decomposed into two separate N-1-step sub-collectives that each move a full Ψ of data.