NaN On Step 300

Training runs smoothly for 300 steps at a learning rate that has worked on this model before. The printed loss goes 0.691, 0.412, 0.221, ..., 0.004, then one step prints nan and every step after it is nan. The loss is hand-written:

p = 1 / (1 + np.exp(-logits))          # float32
loss = -(y * np.log(p) + (1 - y) * np.log(1 - p)).mean()

Which explanation fits this trace?

np.exp overflowed for the large positive logits a confident model produces, so p became inf and every term downstream of it followed.

A confident prediction rounded to exactly 1.0 in float32, so np.log(1 - p) gave log(0) = -inf, and that row's (1 - y) is 0, making the term 0 * -inf, which is nan.

The learning rate is too high: the updates overshot the minimum, the loss diverged past the float32 range, and the overflow surfaced as nan.

The gradients vanished at this depth, and once an update is exactly zero the loss it produces is undefined and stays undefined afterwards.