Grading Against Yourself

A nightly regression suite has scored between 0.93 and 0.95 for four months. Last week a prompt rewrite that support and users both preferred dropped it to 0.71, and the team reverted.

# build_golden_set.py  (run once, in March)
for question in sample_production_queries(500):
    answer = production_model.complete(SYSTEM_PROMPT, question)
    golden.append({"q": question, "expected": answer})

# nightly_eval.py
score = mean(
    judge_similarity(model.complete(SYSTEM_PROMPT, g["q"]), g["expected"])
    for g in golden
)

What is this suite actually measuring?

Real quality, but the sampling temperature was never pinned, so run-to-run variance is being read as a regression.

Similarity to March's outputs, which it has installed as ground truth. Anything that changes how the model phrases a correct answer registers as a regression.

Nothing reliable, because the judge model is too weak to score similarity; the fix is a stronger judge model.

Real quality, but on too small a sample: 500 questions cannot separate a 0.94 from a 0.71 with confidence, so the drop is noise.