Mean Versus Median Intercept

Five observations, no features, so the model is a single constant $c$ :

$y = [1,; 2,; 3,; 4,; 100]$

One fit minimizes $\sum_i (y_i - c)^2$ (ordinary least squares). The other minimizes $\sum_i |y_i - c|$ , which is what QuantileRegressor(quantile=0.5) does.

What constant does each produce?

Squared error gives 22 and absolute error gives 2.5, the midpoint of the two central values.

Squared error gives 22, the sample mean; absolute error gives 3, the sample median. The single outlier drags the least-squares fit past every other observation.

Both give 22. The outlier shifts any fit by the same amount, and the choice of loss only changes how fast the optimizer converges.

Squared error gives 3 and absolute error gives 22, since squaring a large residual shrinks its influence relative to the small ones.