What The Single Head Cannot See

A deep ensemble of $M = 2$ regression networks each outputs a Gaussian $\mathcal{N}(\mu_m(x), \sigma_m^2(x))$ . On one test input:

member	mean $\mu_m$	variance $\sigma_m^2$
1	2.0	1.0
2	6.0	3.0

The ensemble is treated as a uniformly weighted mixture, so $\mu_* = M^{-1}\sum_m \mu_m$ ∗=M−1∑mμm and $\sigma_$ .

What predictive variance does the ensemble report, and what could either network alone have reported?

$\sigma_*^2 = 22.0$ ∗2=22.0, the average of $\sigma_m^2 + \mu_m^2$ across members

$\sigma_*^2 = 2.0$ ∗2=2.0, the average of the members' variances: averaging predictions cannot manufacture variance that neither member reported

$\sigma_*^2 = 6.0$ ∗2=6.0, against $1.0$ or $3.0$ from a single member: the mixture adds $4.0$ of between-member spread to the $2.0$ average within-member variance

$\sigma_*^2 = 4.0$ ∗2=4.0, the spread of the two means, since each member's own $\sigma_m^2$ describes observation noise rather than model uncertainty