Question 847

Gradient At A Confident Mistake

A logistic unit outputs p=σ(z)p = \sigma(z). On one example the label is y=0y = 0 but the model is confidently wrong: p=0.99p = 0.99.

Using σ′(z)=σ(z) (1−σ(z))\sigma'(z) = \sigma(z),(1 - \sigma(z)) from the CS229 derivation:

  • log loss L=−[ ylog⁡p+(1−y)log⁡(1−p) ]L = -[,y\log p + (1-y)\log(1-p),] has ∂L∂z=p−y\dfrac{\partial L}{\partial z} = p - y
  • squared error L=12(p−y)2L = \tfrac{1}{2}(p - y)^2 has ∂L∂z=(p−y) p (1−p)\dfrac{\partial L}{\partial z} = (p - y),p,(1-p)

What are the two gradients at this example?

A

Log loss 0.99 and squared error 0.495, since the squared-error gradient is (p−y)/2(p - y)/2.

B

Log loss 0.99; squared error about 0.0098, roughly 100 times smaller, because the factor p(1−p)=0.0099p(1-p) = 0.0099 multiplies it.

C

Both 0.99: the chain rule applies to both losses in the same way, so the sigmoid factor appears in both and cancels out of the comparison.

D

Log loss about 0.0098 and squared error 0.99: it is the logarithm that saturates when a prediction approaches 1.