Foundations 0/0

Foundations

Transformers

Queries, keys and values, scaled dot-product attention, multiple heads and positional information, assembled into the transformer block that language models are built from. Model cards, serving configs and kernels all assume this vocabulary.

0 / 4 solved

Reading

Papers and essays6

  1. The Illustrated Transformer(opens in a new tab)

    Alammar 2018 Essay 35 min

    A visual walk through the Transformer from Attention Is All You Need: how each word becomes query, key and value vectors, how attention scores are computed and combined across heads, and how the encoder and decoder stacks fit together.

  2. Efficient Estimation of Word Representations in Vector Space(opens in a new tab)

    Mikolov et al. 2013 ICLR 2013 Workshop 40 min

    Two cheap architectures, continuous bag-of-words and skip-gram, that learn word vectors from very large text corpora, plus an analogy test set (king minus man plus woman should land near queen) for measuring them.

  3. Sequence to Sequence Learning with Neural Networks(opens in a new tab)

    Sutskever, Vinyals and Le 2014 NeurIPS 2014 40 min

    Sequence-to-sequence learning: one LSTM reads the source sentence into a fixed-size vector and a second LSTM generates the translation from it, trained end to end. Reversing the source word order improved results markedly.

  4. Neural Machine Translation by Jointly Learning to Align and Translate(opens in a new tab)

    Bahdanau, Cho and Bengio 2015 ICLR 2015 60 min

    Attention for translation: at each output step the decoder takes a learned weighted average of all encoder states, so the source no longer has to fit into a single vector.

  5. Neural Machine Translation of Rare Words with Subword Units(opens in a new tab)

    Sennrich, Haddow and Birch 2016 ACL 2016 40 min

    Byte-pair encoding for word segmentation: start from characters and repeatedly merge the most frequent adjacent pair, giving a fixed vocabulary of subword units that can spell any word.

  6. Attention Is All You Need(opens in a new tab)

    Vaswani et al. 2017 NeurIPS 2017 90 min

    The Transformer: an encoder-decoder built only from multi-head self-attention and feed-forward layers, with positional encodings in place of recurrence, so training runs in parallel across the whole sequence.

Videos5

Neural Networks 3Blue1Brown (Grant Sanderson) 3 videos

Neural Networks: Zero to Hero Andrej Karpathy 2 videos

People to follow1

  • Jay Alammar

    Author of The Illustrated Transformer

    Illustrated guides to the Transformer, BERT and GPT-2 that draw each step of the model, and Hands-On Large Language Models, the book he co-wrote.