Stuck At Zero

A hidden unit computes

$h = \max(0,; w^\top x + b)$

Partway through training the bias has drifted to $b = -6$ , and across the whole current minibatch $w^\top x$ falls in $[-2, 2]$ .

What happens to $w$ and $b$ for this unit on the next gradient step?

The unit outputs a small negative value rather than exactly 0, and that value carries the gradient that lets it recover.

Nothing: the pre-activation is negative for every example, so $\partial h / \partial(w^\top x + b) = 0$ , the gradient reaching $w$ and $b$ is exactly zero, and the unit stays dead unless another path changes its inputs.

They move a little: the gradient is small but nonzero on the flat side, so the unit recovers slowly over many steps.

They receive the full upstream gradient, because the rectifier passes gradient through unchanged and only scales the forward value.