1 / 5 Evals, Guardrails & LLM Observability

Two Runs Disagree

You run 500 pairwise judgments where an LLM judge picks the better of two responses. When you rerun the same 500 pairs with the two responses presented in the opposite order, the judge's pick flips on about 40 percent of them. What is happening and what is the standard fix?

A

The judge is weaker than the models it compares; move to a larger judge and keep one order

B

Position bias; judge every pair in both orders and count a win only when both orders agree

C

Verbosity bias; truncate both responses to the same token length before judging

D

Sampling noise from a nonzero temperature; set temperature to 0 and judge each pair once