Post-training 0/0

Post-training

RLHF and DPO

Training a model on human preferences, either with a reward model fitted to ranked pairs and PPO against it, as in InstructGPT, or with Direct Preference Optimization, which needs no reward model. This step turns a base model into an assistant.

Reading

  1. Training language models to follow instructions with human feedback(opens in a new tab)

    Ouyang et al. 2022 NeurIPS 2022 90 min

    InstructGPT: fine-tune GPT-3 on human-written demonstrations, train a reward model on human rankings of outputs, then optimize the model against that reward with reinforcement learning (RLHF).

  2. Direct Preference Optimization: Your Language Model is Secretly a Reward Model(opens in a new tab)

    Rafailov et al. 2023 NeurIPS 2023 60 min

    DPO: fine-tune a language model on pairs of preferred and rejected responses with a simple classification loss, which optimizes the same objective as RLHF without training a separate reward model or sampling from the model during training.