Perturbed Ridge

Suppose that you perform OLS and Ridge regression using the same data matrix, $X$ . Denote the fitted coefficients as $\hat{\beta}$ and $\hat{\beta}$ {Ridge} $\hat{β}_{R i d g e}$ , respectively. Suppose that you then add some random noise to $X$ and perform each regression again, getting fitted coefficients $\hat{\beta}^{$ and $\hat{\beta}^{$ }{Ridge} $\hat{β}_{R i d g e}^{*}$ . In general, which do you expect to be larger, $||\hat{\beta}$ {OLS} - \hat{\beta}^{}{OLS}|| $∣∣ \hat{β}_{O L S} - \hat{β}_{O L S}^{*} ∣∣$ or $||\hat{\beta}$ {Ridge} - \hat{\beta}^{}_{Ridge}|| $∣∣ \hat{β}_{R i d g e} - \hat{β}_{R i d g e}^{*} ∣∣$ ?

The OLS change is always zero if the noise has mean zero.

In general, the OLS change is expected to be larger because ridge regularization stabilizes the inverse, especially under multicollinearity.

In general, the ridge change is always larger because ridge shrinks coefficients.

They must be exactly equal because both regressions use the same data matrix.