A critical methodological consideration in machine learning for predicting gallbladder cancer metastasis

Dear Editor, We enjoyed reading the study by Liu et al on the development of an extreme gradient boosting (XGB) model to predict distant metastases (DM) of gallbladder cancer[1]. The idea of a web-based calculator for clinical decision support is nice. We will also mention some important methodological problem of class imbalance that may influence the interpretation of the reported performance of the model and its readiness to translation in clinical settings. The authors correctly used SMOTE on the training set to address the class imbalance, however, the subsequent results appear contradictory. Figure 3C and related text show that the optimal XGB model without SMOTE achieved average AUC 0.99 cross validation, while the final model performance shown in Table 4 and Fig 4 (AUC 0.885 on internal test set) is probably from the model trained with SMOTE, and this is not reconcilable. The general rule for imbalanced data is that resampling methods like SMOTE should be applied only to the training fold during cross-validation, or to the training set before hold-out validation, so that the test data remain untouched to simulate real-world deployment[2]. Reporting metrics from a model (even one trained on artificially balanced data) that is evaluated on an artificially balanced test set can lead to a significant optimism bias[3]. The very high precision (0.968) and specificity (0.968) on imbalanced internal test set (Table 4) seem very high for a clinical outcome and warrant further investigation into potential data leakage or overfitting during the feature selection/model tuning process, which may have been influenced by the synthetic samples. For a prediction model to be credible, its final performance must be evaluated on a natural, imbalanced dataset that reflects the true population prevalence[4]. We suggest authors clarify the pipeline: which final model (trainable with or without SMOTE) is shown in results and powers the web calculator? A more transparent evaluation would involve reporting the performance of the SMOTE-trained model solely on the natural, imbalanced test set, thereby avoiding confusion with the cross-validation results obtained without SMOTE. Machine learning can be very promising for cancer but clinical applications require rigorous, transparent and reproducible methods[5]. We strongly suggest the authors clarify this point, as it is essential for evaluating the true discriminatory ability of the model and its safe integration into clinical workflows to guide personalized treatment for gallbladder cancer patients[6].