Sample size determination for external validation of risk models with binary outcomes using the area under the ROC curve

Abstract External validation is critical in risk prediction model development to assess the model’s performance on independent datasets before clinical application. Sample size determination (SSD) is a pivotal aspect for external validation, ensuring sufficient precision, adequate power, appropriate ethical considerations, and affordable resource allocation. SSD for external validation of a risk model with binary outcomes can be performed targeting the most commonly used evaluation criterion, the area under the receiver operating characteristic (AUROC) curve, for two study designs: one focusing on margin-of-error and another on hypothesis testing. Sample sizes for external validation are estimated using various AUROC variance estimators. In this article, we conduct extensive simulations to compare the performance of different AUROC-based SSD methods for external validation under the two study designs. These simulations involve models developed using various machine learning (ML) algorithms, different sets of predictors, and data generated under diverse distributions. The simulation results show significantly different sample size estimates based on the chosen SSD methods. Based on empirical evidence from these simulations, we recommend the Hanley and McNeil method or the rules-of-thumb approach for margin-of-error–based designs, given their balance between reliability and feasibility. For hypothesis testing designs, the Hanley and McNeil method demonstrates superior operating characteristics and is recommended for achieving adequate power. We also demonstrate the application of these AUROC-based SSD methods in the external validation of a risk prediction model for forecasting 1-year overall survival (OS) in pediatric patients who underwent allogeneic hematopoietic cell transplantation (alloHCT). Our findings highlight the importance of selecting appropriate SSD methods for external validation and provide practical guidance for implementing SSD in clinical prediction model validation.