The Slow Roll

An engineer adds momentum to plain SGD when training a CNN and notices training loss descends faster and oscillates less across noisy minibatches. What is the core mechanism by which momentum achieves this?

It periodically resets the optimizer state to escape saddle points

It accumulates an exponentially weighted moving average of past gradients and uses that as the update direction, smoothing out noise and building speed along consistent gradient directions

It rescales the learning rate for each parameter based on the magnitude of its historical gradients

It clips each minibatch gradient to a fixed norm before applying the update, preventing any single noisy batch from causing a large step