Risk prediction for individual patients and the pitfalls of selecting an optimal prediction model: do not judge a model by its c-statistic

Abstract Researchers often report the area under the curve (AUC/c-statistic), Brier-index (BI), and explained variation (R 2 ) to assess risk prediction model performance. This may be inappropriate for scores to estimate individual patient risks because population characteristics like risk score distributions may influence these performance metrics. This study assessed performance values for a fully accurate risk score under different distributions. A risk score ranging 0-100% was simulated for 100,000 individuals with 1000 bootstraps under six distributions: normal, uniform, bimodal, extreme bimodal, uniform ascending and uniform descending. Outcomes were simulated with full accuracy: 1% of individuals with a score of 0.01 had an outcome, 2% with 0.02 etc. Values for the AUC, BI, BI scaled (BS), and R 2 were calculated, and calibration plots made. Sensitivity analyses included narrowing risk score range and reducing score granularity. For a normally distributed score, performance indices did not exceed values generally considered poor (AUC = 0.67, BI = 0.23, BS = 0.08, R 2 = 8%). Values were best for an extreme bimodal distribution (AUC = 0.93, BI = 0.10, BS = 0.61, R 2 = 61%). Performance indices for the other distributions were moderate (AUC = 0.80–0.83, BI = 0.17–0.18, BS = 0.25–0.33, R 2 = 25–33%). Calibration plots were perfect for all distributions. Values worsened with narrowing score ranges. Reducing granularity had marginal effects. Risk prediction model performance values depend strongly on risk distributions. They may not exceed values generally considered poor/moderate in many common risk distributions. When assessing patient risk prediction models, investigators and clinicians interested in the accuracy of risk estimations may prioritize calibration and net benefits over commonly used metrics like the AUC and BI.