Where The Decay Goes

Two optimizer inner loops, identical except for where the decay term sits:

# variant A
g = grad(loss, w) + wd * w
m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g * g
w -= lr * m_hat / (sqrt(v_hat) + eps)

# variant B
g = grad(loss, w)
m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g * g
w -= lr * (m_hat / (sqrt(v_hat) + eps) + wd * w)

Which is AdamW, and what is the observable consequence of the difference?

B, and the difference is that AdamW applies decay only on steps where the gradient is close to zero, which is why it acts as a pruning signal.

B. In A the decay term passes through the division by $\sqrt{\hat v}$ , so a weight with a large gradient history is decayed less than a quiet one, making the effective regularization strength per-parameter and dependent on the gradient statistics.

A. Decoupling means folding the decay into the gradient before the moment estimates, so that momentum smooths it along with everything else.

They differ only by a rescaling of wd by lr, so the choice shifts where the good hyperparameter range sits but not the optimum reached.