2 / 5 Evals, Guardrails & LLM Observability

Live Fire Rehearsal

Before exposing anyone to a new retrieval and model configuration, you want to see how it behaves on the current real query mix. Which setup gives you that with no user-visible risk?

A

Expose the new stack behind an internal feature flag and have the engineering team use it for a week before launch

B

Mirror a copy of live requests to the new stack, discard its responses, and diff them offline against production

C

Route 5 percent of users to the new stack and compare their session metrics against the remaining 95 percent

D

Replay last month's logged prompts through both stacks and diff the two sets of outputs offline