Decoupled Discipline

With plain SGD, adding an L2 penalty to the loss is mathematically equivalent to applying weight decay directly to the weight update. Why does this equivalence break down for Adam, motivating AdamW's decoupled weight decay?

Adam's momentum term cancels out any L2 penalty added to the loss before the update is applied

Adam's per-parameter adaptive learning rates rescale the L2 gradient term itself, so weight decay ends up applied unevenly across parameters unless it is decoupled from the gradient-based update

Adam does not use gradients at all, so an L2 penalty added to the loss has no effect on the update

L2 regularization only works with a learning rate schedule, and Adam is incompatible with schedules