6 / 9 Ensembles & Neural-Net Basics

Ten Layers of Squash

CS229 derives σ′(z)=σ(z) (1−σ(z))\sigma'(z) = \sigma(z),(1 - \sigma(z)) for the logistic sigmoid.

In a 10-hidden-layer stack of sigmoid units, backpropagation multiplies one σ′\sigma' factor per layer on its way back to the first layer. Ignoring the weight matrices, what is the largest value σ′\sigma' can take, and the largest possible product of the ten factors?

A

σ′\sigma' peaks at 0.5 (the value of σ(0)\sigma(0)), so the product is 0.510≈10−30.5^{10} \approx 10^{-3}.

B

σ′\sigma' peaks at 0.25 (at z=0z = 0), so the product is at most 0.2510≈10−60.25^{10} \approx 10^{-6}: even in the best case, the gradient reaching layer 1 is about a millionth of the gradient at the output.

C

σ′\sigma' peaks at 0.25, and the ten layers contribute 10×0.25=2.510 \times 0.25 = 2.5.

D

σ′\sigma' peaks at 1.0, since a function bounded by 1 has a derivative bounded by 1, so the product is about 1 and depth costs nothing.