Correcting a Classifier When Class Frequencies Change: Prior Adjustment by EM and by Confusion-Matrix Inversion on Three UCI Data Sets, and When the Correction Does Not Help

A classifier trained where 40% of cases are positive will give poorly calibrated probabilities where 10% are. If only the class frequencies change (label shift), Bayes' rule gives an exact correction once the new frequency is known, and the frequency can be estimated from unlabelled data. I compare two estimators, the EM method of Saerens, Latinne and Decaestecker and black-box shift estimation (BBSE) by confusion-matrix inversion, with no correction and with the correction at the true frequency, using logistic regression on three UCI data sets and test sets resampled to seven target prevalences from 0.1 to 0.9 (50 replicates each). On the MAGIC gamma telescope data, averaged over targets, correction cuts log loss from 0.5204 to 0.3704 (EM) and 0.3702 (BBSE), against 0.3697 with the true frequency, and cuts the error in the estimated prevalence from 0.1795 to 0.0100 and 0.0092. On spambase the log loss falls from 0.2505 to about 0.195. On the banknote data, where the classifier is nearly perfect, the gain is small (0.0570 to 0.0466–0.0485) and at high target prevalences accuracy is slightly lower after correction. Both estimators come within 0.0018 of the oracle log loss on average. The correction helps most when the shift is large and the classifier's probabilities are uncertain.