Stuck At Zero

A hidden unit computes

$h = \max(0,; w^\top x + b)$

Partway through training the bias has drifted to $b = -6$, and across the whole current minibatch $w^\top x$ falls in $[-2, 2]$.

What happens to $w$ and $b$ for this unit on the next gradient step?