One Layer Upstream Of The Loss

SimCLR trains an encoder $f$ producing $h$ (2048-d) and a projection head $g$ producing $z = g(h)$ (128-d); the NT-Xent contrastive loss is applied to $z$ . After pretraining the head is thrown away and downstream models use $h$ .

Two probes run on the frozen network: a linear classifier on $h$ against one on $z$ , and a probe trained to recover which augmentation was applied (for instance the rotation) from $h$ against from $z$ .

Which outcome does the paper report, and what mechanism produces it?

The two score the same under linear evaluation; the head is an optimization device that conditions the loss surface and leaves the representation unchanged

$z$ beats $h$ , because the projection strips nuisance directions the encoder leaves behind and the loss is computed in that cleaner space

$h$ beats $z$ by more than 10 points on linear evaluation, and the rotation probe reads 67.6% from $h$ against 25.6% from $z$ : the loss trains $z$ to be invariant to the augmentations

$h$ beats $z$ only because it is 2048-d against 128-d; widening the projection output to 2048 closes the gap