Sixty Five Percent Consistent

The MT-Bench authors measured how often an LLM judge gives the same verdict when the two candidate answers are swapped:

judge	consistency under swap	favors the first answer	favors the second
Claude-v1	23.8%	75.0%	0.0%
GPT-3.5	46.2%	50.0%	1.2%
GPT-4	65.0%	30.0%	5.0%

You adopt their conservative protocol: judge each pair twice, once in each order, and declare a win only if the same answer wins both times, otherwise record a tie. Your judge behaves like the GPT-4 row. You run 800 pairs.

What happens to your statistical power?

None come back tied. Swapping cancels position bias, so both orders agree by construction.

About 280 pairs tie purely from order sensitivity, leaving 520 decisive, so the minimum detectable difference widens by $\sqrt{800/520} \approx 1.24$ .

About 280 come back tied, and the surviving 520 are order-independent, so power is unchanged: you have simply removed the noise.

About 520 come back tied, because 65 percent consistency means the judge agrees with human labels only 65 percent of the time.