Foundations 0/0

Foundations

Training deep networks

Convolutional, recurrent and residual networks, and the normalisation, regularisation and loss choices that let deep networks train and generalise. Residual connections and layer normalisation carry straight into the transformer.

0 / 14 solved

Reading

Papers and essays5

  1. A Recipe for Training Neural Networks(opens in a new tab)

    Karpathy 2019 Essay 30 min

    A step-by-step process for training a network without silent bugs: inspect the data by hand, build the full training and evaluation pipeline around a simple model, overfit a small batch, then add regularization and tune, checking each step before moving on.

  2. ImageNet Classification with Deep Convolutional Neural Networks(opens in a new tab)

    Krizhevsky, Sutskever and Hinton 2012 NeurIPS 2012 45 min

    AlexNet: an eight-layer convolutional network with 60 million parameters, trained on two GPUs with ReLU activations, dropout and data augmentation, that won the ILSVRC 2012 image classification contest with a top-5 error of 15.3 percent against 26.2 percent for the second-best entry.

  3. Deep Residual Learning for Image Recognition(opens in a new tab)

    He et al. 2016 CVPR 2016 50 min

    Residual connections: each block learns a change that is added to its input, which made networks of up to 152 layers trainable and won the ILSVRC 2015 classification task.

  4. Long Short-Term Memory(opens in a new tab)

    Hochreiter and Schmidhuber 1997 Neural Computation 90 min

    The LSTM cell: a memory unit whose contents carry from step to step through a self-connection of weight 1, with multiplicative input and output gates deciding what to write and what to read, so error signals can flow across more than 1,000 time steps without vanishing.

  5. The Unreasonable Effectiveness of Recurrent Neural Networks(opens in a new tab)

    Karpathy 2015 Essay 40 min

    Character-level recurrent networks trained on Shakespeare, Wikipedia, LaTeX and the Linux kernel source generate convincing text in each style one character at a time, and the post visualizes individual neurons that track quotes, line length and indentation.

Videos4

Neural Networks: Zero to Hero Andrej Karpathy 4 videos

Links2

People to follow1

  • Jeremy Howard

    Founder of fast.ai, CEO of Answer.AI

    fast.ai's free Practical Deep Learning course is for people with some coding experience and needs no special hardware.