7 / 10 Regularization & Loss Techniques

Four Thirds Variance

Ioffe and Szegedy's Algorithm 2 turns a trained BatchNorm layer into a fixed transform at inference:

Var[x]←mm−1 EB[σB2],y=γVar[x]+ϵ x+(β−γ E[x]Var[x]+ϵ)\mathrm{Var}[x] \leftarrow \frac{m}{m-1}, \mathrm{E}{\mathcal{B}}[\sigma{\mathcal{B}}^2], \qquad y = \frac{\gamma}{\sqrt{\mathrm{Var}[x]+\epsilon}}, x + \left(\beta - \frac{\gamma,\mathrm{E}[x]}{\sqrt{\mathrm{Var}[x]+\epsilon}}\right)

For one channel, trained with minibatch size m=4m=4: the averaged minibatch variance EB[σB2]=0.75\mathrm{E}{\mathcal{B}}[\sigma{\mathcal{B}}^2] = 0.75, E[x]=1\mathrm{E}[x] = 1, γ=2\gamma = 2, β=0.5\beta = 0.5, ϵ≈0\epsilon \approx 0.

What does this layer compute at inference?

A

Var[x]=0.5625\mathrm{Var}[x] = 0.5625, so y≈2.67x−2.17y \approx 2.67x - 2.17

B

Var[x]=1.0\mathrm{Var}[x] = 1.0, so y=2x−1.5y = 2x - 1.5

C

Var[x]=0.75\mathrm{Var}[x] = 0.75, so y≈2.31x−1.81y \approx 2.31x - 1.81

D

Var[x]=1.0\mathrm{Var}[x] = 1.0, so y=2x+0.5y = 2x + 0.5