Question 786

The Understudy Model

A team wants to validate a new recommendation model on live production traffic before it can affect any user, while also tracking whether its prediction distribution starts diverging from the incumbent model over time as a trigger for retraining. Which deployment pattern fits, and what should they monitor to decide when a retrain is warranted?

A

A/B testing, since only real user outcomes (not distributions) can justify a retrain decision

B

Shadow deployment, monitoring divergence (e.g., PSI or KL divergence) between the new model's prediction distribution and the live model's over time

C

Canary deployment, since routing a small percent of real user traffic is required to observe prediction drift

D

Blue-green deployment, since it eliminates the need to compare prediction distributions at all