3 / 5 Evals, Guardrails & LLM Observability

Nothing Deployed That Day

Your service calls the model by a floating alias rather than a dated snapshot id. Overnight the nightly regression suite drops from 91 percent to 83 percent. No code, prompt, retrieval index, or eval data changed. What is the likely cause and the right response?

A

The suite has accumulated stale cases, so retire the failing examples and reset the baseline to 83 percent

B

The alias now resolves to a newer model version; pin to a dated snapshot and rebaseline the suite before adopting it

C

The judge model drifted, so regenerate the golden dataset's reference answers with the current judge and rebaseline the suite

D

The prompt cache expired overnight, so warm the cache before the nightly run and rerun the suite