Question 509

Why can ECE comparisons be unstable?

Two researchers evaluate the same classifier and obtain substantially different Expected Calibration Error values. Which explanation is most technically plausible?

A

ECE depends on estimation choices such as binning, sample size, and whether only maximum confidence or all class probabilities are evaluated

B

ECE is a strictly proper scoring rule and therefore changes whenever accuracy is unchanged

C

ECE is uniquely determined by the confusion matrix, so the discrepancy proves one implementation is incorrect

D

ECE is invariant to bin edges but sensitive only to class prevalence