Why can deep ensembles improve uncertainty estimates?

Ensemble variance is guaranteed to be calibrated even when every member is trained on identical corrupted labels

Their averaged output is mathematically guaranteed to equal the exact Bayesian posterior predictive distribution

Different initializations and training trajectories can produce diverse predictive functions whose disagreement carries information about model uncertainty

Averaging eliminates aleatoric uncertainty while retaining only epistemic uncertainty