Evaluation metrics under class imbalance in machine learning: A methodological analysis

Abstract Class imbalance alone does not identify an appropriate evaluation metric, because common measures target different prediction objects. This study combines a five-dataset decision audit with controlled population analyses, and separates what follows from established identities from what the constructions and observed classifier outputs add. With class conditional rates held fixed, Balanced Accuracy, the geometric mean, and Youden’s J stay constant across prevalence, while the Matthews correlation coefficient keeps its known prevalence dependence and tends to zero with the square root of prevalence in the rare class limit when specificity is strictly between zero and one. In a symmetric Gaussian model, the three class conditional objectives share a posterior threshold equal to the prevalence, while the F1 score, the Matthews correlation coefficient, and Accuracy select higher ones. Under unequal variance and a logistic score family, the shared optimal posterior cutoff still equals the prevalence, but its acceptance set need not be one raw score interval and the geometric mean leaves it. For calibrated deployment probabilities and constant error costs, the cost optimal posterior threshold is one over one plus the false negative to false positive cost ratio. In the ranking construction, the area under the receiver operating characteristic curve keeps its model preference at every prevalence, while the area under the precision recall curve reverses that ranking below prevalence 0.0253. Across five public datasets, fitted candidates are compared under screening, review capacity, probability forecasting, and cost objectives. In the pulsar case, screening detects 296 of 328 test positives with 50 false positives, whereas a 5% review budget retrieves 178 positives among 179 reviewed cases. Validation sensitivity targets do not guarantee test sensitivity, and chronological marketing data expose a calibration-induced ranking reversal. Evaluation should follow the prediction object and the decision question, stating prevalence, threshold policy, cost, probability quality, averaging, and uncertainty.