Huber at a Residual of Ten

PyTorch's HuberLoss with delta = 1 is, per element,

$\ell(a) = \begin{cases} 0.5,a^2 & |a| < \delta \ \delta,(|a| - 0.5,\delta) & \text{otherwise} \end{cases}$

One outlier in the batch has residual $a = 10$ ; a typical point has $a = 0.5$ . Compare $\ell$ and $d\ell/da$ against plain squared error $0.5a^2$ at both residuals.

Outlier: Huber $50$ with gradient $10$ , the same as squared error, since $\delta = 1$ only changes the loss for residuals below 1.

Outlier: Huber $9.5$ with gradient $1$ , squared error $50$ with gradient $10$ , so the outlier pulls the fit ten times less hard under Huber. Typical point: both give $0.125$ with gradient $0.5$ , identical, because $|a| < \delta$ .

Outlier: Huber $10$ with gradient $1$ , squared error $50$ with gradient $10$ . Typical point: identical under both losses.

Outlier: Huber $9.5$ with gradient $10$ : the linear branch caps the loss value, but the gradient still scales with the residual.