Question 471

The Additive One

In a plain 50-layer stack each layer contributes a Jacobian factor to the backward pass. If every factor has spectral norm about 0.6, the gradient reaching the first layer is scaled by roughly 0.650≈8×10−120.6^{50} \approx 8 \times 10^{-12}.

He et al. write the residual case with identity shortcuts as

xL=xl+∑i=lL−1F(xi),∂E∂xl=∂E∂xL(1+∂∂xl∑i=lL−1F(xi))\mathbf{x}_L = \mathbf{x}l + \sum{i=l}^{L-1} \mathcal{F}(\mathbf{x}_i), \qquad \frac{\partial \mathcal{E}}{\partial \mathbf{x}_l} = \frac{\partial \mathcal{E}}{\partial \mathbf{x}_L}\left(1 + \frac{\partial}{\partial \mathbf{x}l}\sum{i=l}^{L-1}\mathcal{F}(\mathbf{x}_i)\right)

What does this form change?

A

The residual branch is bypassed during backpropagation, so gradients travel only along the identity path and reach the first layer unattenuated.

B

The block contributions are summed inside one factor rather than multiplied across 50 factors, so the gradient at block ll is the gradient at block LL plus a correction. The additive 1 is a direct path that survives however small the residual branches' derivatives get, and it can only cancel if that sum equals exactly −1-1, which will not hold across a whole mini-batch.

C

Each layer's Jacobian gains an added identity, so the factors become about 1.6 and the gradient explodes as 1.6501.6^{50} instead of vanishing.

D

The shortcut normalizes activations so that every Jacobian's spectral norm is driven to exactly 1, which is what keeps the product from shrinking.