Assessing the capability of supervised classifiers to recover the true underlying probabilities

In classification, a key objective is to produce estimated probabilities that approximate the true underlying probabilities. However, evaluating the quality of these estimates remains challenging since the true probabilities are unobservable.In this study, we simulate datasets with controlled true distribution shapes to analyze how well different models preserve these distributions. We also assess for these models how well standard calibration metrics (Expected Calibration Error, Brier Score, Log-Loss) reflect the quality of the predicted probabilities.Our results show that while models like logistic regression, neural networks, and SVM provide good probability estimates, others like random forests and naive Bayes tend to distort the true probabilities. Importantly, standard calibration metrics such as ECE fail to detect these distortions. In contrast, loss-based metrics such as Log-Loss and Brier Score, especially when decomposed into epistemic and refinement components, offer more informative assessments. These findings highlight that a classifier may appear well calibrated according to ECE, even when its predicted probabilities are misaligned with the true underlying probabilities.This study emphasizes that when accurate probability estimation is a primary modeling goal, practitioners should rely on loss-based evaluations for comparing models to choose the best one in calibration rather than solely on ECE, and preference should be given to models that better preserve the shape of the true probability distribution.