Evals 0/1

Evals

LLM judges

Using one model to grade another model’s outputs against a rubric, and measuring how often it agrees with human graders. Judges make open-ended outputs measurable, but their position, length and self-preference biases have to be checked.

Reading

Papers and essays1

  1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena(opens in a new tab)

    Zheng et al. 2023 NeurIPS 2023 Datasets and Benchmarks 50 min

    Using a strong model to grade other models' answers, checked against human votes on MT-Bench and Chatbot Arena. GPT-4 as judge agreed with humans over 80 percent of the time, about as often as humans agree with each other.

People to follow1