Evals, Guardrails & LLM Observability

Offline evals, LLM-as-judge, golden datasets, A/B and shadow traffic, tracing, and drift after model upgrades.

0 / 5 solved

Question Tags Difficulty Success

1. Two Runs Disagree

llm-as-judgejudge-bias +1

Medium —

2. Live Fire Rehearsal

shadow-trafficonline-eval +1

Medium —

3. Nothing Deployed That Day

model-version-pinningquality-drift +1

Hard —

4. Grading Against Yourself

golden-datasetregression-suite +3

Hard —

5. Sixty Five Percent Consistent

llm-as-judgeposition-bias +2

Hard —

Learning Resources

OpenAI Evals

Up next

5 of 5 questions here are still unsolved.

Next module Prompt Injection & AI Security