Coverage Drifts Away

A team calibrates a split conformal predictor on 2,000 January-to-June insurance claims at $\alpha = 0.10$ , then serves it unchanged for the rest of the year. Measured coverage on served requests: 91% in July, 88% in September, 79% in December. Over the same months the base model's own accuracy drifted down.

What does this pattern indicate?

Coverage fell because the base model degraded: conformal intervals inherit the accuracy of the predictor they wrap

Nothing is broken: conformal coverage is distribution-free, so the guarantee holds whatever distribution the December claims follow

The calibration set is too small: the guarantee only holds up to a slack of $1/(n+1)$ , and 2,000 points leave room for an 11-point shortfall

Exchangeability between the calibration and test points broke as the claim distribution moved through the year, and the coverage shortfall tracks how far the later data drifted