The Sharp Coordinate Sets the Step

Two features are on wildly different scales, so the squared-error surface is

$J(w) = 50w_1^2 + 0.5w_2^2$

Gradient descent uses one learning rate for both coordinates, $w \leftarrow w - \eta \nabla J(w)$ , which works out per coordinate to

$w_1 \leftarrow (1 - 100\eta),w_1 \qquad w_2 \leftarrow (1 - \eta),w_2$

What is the FULL range of $\eta$ over which both coordinates converge, and what does that range cost?

$0 < \eta < 0.01$ , because a step must stay below one over the largest curvature.

$0 < \eta < 0.02$ , set by $w_1$ . At the largest usable step $w_2$ keeps 98% of its value each iteration, so it needs about 34 iterations just to halve.

$0 < \eta < 2$ , set by $w_2$ : the flat direction is the one that limits stability.

Any positive $\eta$ works, because gradient descent rescales each coordinate by its own curvature automatically.