Does the classifier beat chance? A Monte Carlo simulation procedure for the empirical estimation of performance baselines in imbalanced datasets with annotated example

Objective: the evaluation of binary classifiers in imbalanced datasets requires an empirical reference of the expected performance under random classification in order to correctly interpret observed metrics. Without that reference, it is not possible to determine whether a classifier's performance reflects genuine discriminative capacity over chance or simply the structural effect of class imbalance. This paper proposes and demonstrates a Monte Carlo simulation procedure to empirically estimate that reference with its full distribution and 95% central interval for any benchmark dataset with known prevalence.Design and methodology: the procedure generates, through N parametrizable iterations, empirical distributions of eight binary classification metrics (recall, specificity, precision, NPV, F1, F2, Balanced Accuracy, and MCC) under two reference scenarios: random classification with uniform distribution across categories (Scenario A) and random classification informed by the actual distribution of the reference coder (Scenario B). The central 95% interval of the simulated null distribution is obtained using the empirical percentile method, without assumptions about the shape of the distribution. The implementation is provided in R with an alternative version in Python and an Excel data entry template. The procedure is demonstrated on the corpus of Alfalla-Luque et al. (2023), a systematic review on supply chain agility with 593 references and a prevalence of 0.118.Results: the simulation on the example corpus shows the difference between the theoretical point expected value and the 97.5th percentile of the empirical distribution for all evaluated metrics. For recall under uniform random classification, the correct operational threshold is 0.614, which is 23% more demanding than the theoretical value of 0.5. For MCC, the 97.5th percentile under chance is 0.083 and not 0. The comparison between scenarios reveals that Balanced Accuracy and MCC are insensitive to the asymmetry distribution, while recall, F1, and F2 are highly sensitive to it.Value and implications: the procedure is applicable in the context of title and abstract screening in systematic literature reviews, where the structural imbalance of corpora means that the random classification cut-off value is always corpus-specific and never generalizable across reviews (like financial fraud detection, clinical diagnosis of low-prevalence conditions, rare event classification in industrial time series, and automated content moderation on digital platforms). The code, materials, and example corpus are available in a public repository.