Optimistic By Three Points

Two pipelines are evaluated on the same imbalanced dataset, both reporting balanced accuracy:

pipeline	cross-validation score	held-out test score
resample the whole dataset, then cross-validate the classifier	$0.724 \pm 0.042$	$0.698 \pm 0.014$
resample inside each training fold via a `Pipeline`	$0.732 \pm 0.019$	$0.727 \pm 0.008$

What do these four numbers tell you?

The validation folds should be resampled too, so that cross-validation and the held-out test measure the same class distribution.

The first pipeline's cross-validation score is optimistic and its variance is more than double, because synthetic points interpolated from samples that later land in the validation fold appear in the training folds. The second keeps cross-validation and held-out test agreeing, which is the property you actually want from a model-selection procedure.

The first pipeline is acceptable: 0.724 and 0.698 are within one standard deviation of each other, so the difference is noise.

The first pipeline underfits. Synthetic samples add noise that the model has to average over, which is why its test score is lower.