Limitations of Imbalanced Data Resampling Methods: The Need to Evaluate Calibration and Overall Performance

Prediction models are useful in various fields and require models with excellent performance, especially in the health and medical fields.However, the class imbalance problem occurs frequently in clinical data, and the imbalanced data problem causes the performance of the prediction model to deteriorate.Research is steadily being conducted to solve the imbalanced data problem to improve the performance of prediction models.Most of these studies focus on discrimination.However, even if the discrimination is excellent, validation of whether the estimated probability from the model is a reliable value is insufficient.To supplement this issue, calibration and the overall model performance were evaluated at the same time.The evaluation of the overall model performance considers discrimination and calibration at the same time, and Calibration compares the estimated probability in the model to the observed value to evaluate the degree of agreement between the two values.In this paper, simulation and real data analysis were performed to evaluate and compare several resampling methods proposed to solve the imbalanced data problem from the perspective of discrimination, calibration, and the overall model performance.As a result, it was shown that discrimination did not guarantee the accuracy of the predicted values, and it was shown that a validation process of the predicted values, such as calibration, is necessary.