Foundations 0/0

Foundations

Neural networks

A network as layers of weighted sums and nonlinear activations, trained by backpropagation with an optimizer such as SGD with momentum or Adam. Every model in the later tracks is trained this way.

0 / 10 solved

Reading

Papers and essays2

  1. Learning representations by back-propagating errors(opens in a new tab)

    Rumelhart, Hinton and Williams 1986 Nature 30 min

    Backpropagation for networks with hidden units: run the input forward, pass the error derivative backward through each layer with the chain rule to get the gradient for every weight, and adjust the weights by gradient descent. The hidden units learn useful internal features on tasks such as detecting symmetry and learning family relationships.

  2. Adam: A Method for Stochastic Optimization(opens in a new tab)

    Kingma and Ba 2015 ICLR 2015 40 min

    Adam: an optimizer that keeps running averages of each parameter's gradient and squared gradient, corrects their bias toward zero early in training, and scales each step by their ratio, so every parameter gets its own step size.

Videos6

Neural Networks 3Blue1Brown (Grant Sanderson) 4 videos

Neural Networks: Zero to Hero Andrej Karpathy 2 videos

Links3

People to follow1