Foundations 0/0

Foundations

How LLMs are built

How a transformer becomes a chat model: tokenisation, pretraining on next-token prediction, scaling laws, then instruction tuning and preference training. The agent, eval, post-training and inference tracks all work with this model.

Reading

Papers and essays9

  1. What Is ChatGPT Doing … and Why Does It Work?(opens in a new tab)

    Wolfram 2023 Essay 120 min

    A long explanation, for readers without a machine learning background, of how ChatGPT produces text: it repeatedly picks a next token from probabilities computed by a trained neural network, and Wolfram builds up to that through embeddings, training and the Transformer with small examples.

  2. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding(opens in a new tab)

    Devlin et al. 2019 NAACL 2019 60 min

    BERT: pre-train a Transformer encoder on unlabeled text by predicting masked words and whether two sentences follow each other, then fine-tune the same model with one small output layer per task.

  3. Language Models are Unsupervised Multitask Learners(opens in a new tab)

    Radford et al. 2019 50 min

    GPT-2: a Transformer language model of up to 1.5 billion parameters trained on WebText, 40 GB of text from pages linked on Reddit, that answered questions, translated and summarized with no task-specific training, by treating each task as text to continue.

  4. Scaling Laws for Neural Language Models(opens in a new tab)

    Kaplan et al. 2020 90 min

    Language model loss falls as a power law in model size, dataset size and training compute, with some trends spanning more than seven orders of magnitude, while width and depth matter little within a wide range.

  5. Language Models are Few-Shot Learners(opens in a new tab)

    Brown et al. 2020 NeurIPS 2020 120 min

    GPT-3, a 175-billion-parameter decoder-only model that performs many tasks from a few examples placed in the prompt, with no gradient updates (in-context learning).

  6. Training Compute-Optimal Large Language Models(opens in a new tab)

    Hoffmann et al. 2022 NeurIPS 2022 75 min

    For a fixed compute budget, parameters and training tokens should grow in equal proportion. Chinchilla, a 70B model trained on 4 times more data with Gopher's compute budget, outperformed the 280B Gopher.

  7. The Llama 3 Herd of Models(opens in a new tab)

    Grattafiori et al. 2024 240 min

    Meta's report on Llama 3: dense Transformers of up to 405B parameters with grouped-query attention and a context window of up to 128K tokens, pre-trained on 15.6T tokens and post-trained with supervised fine-tuning, rejection sampling and DPO.

  8. The Bitter Lesson(opens in a new tab)

    Sutton 2019 Essay 10 min

    Sutton's argument from 70 years of AI research that general methods which use more computation, chiefly search and learning, end up beating methods built on human knowledge of the domain, with chess, Go, speech recognition and computer vision as his examples.

  9. Software 2.0(opens in a new tab)

    Karpathy 2017 Essay 15 min

    Karpathy's argument that neural networks are a new way to write software: instead of coding the behavior, engineers collect a dataset and choose an architecture, and optimization finds the program, which is stored as weights.

Videos4

Building with LLMs Various speakers 2 videos

Neural Networks 3Blue1Brown (Grant Sanderson) 1 video

Neural Networks: Zero to Hero Andrej Karpathy 1 video

People to follow2

  • Sebastian Raschka

    Author of Build a Large Language Model (From Scratch)

    Code-first explanations of LLM architectures and training, written so you can implement the pieces yourself.