Beyond the ROC curve: The U-smile for transparent and interpretable variable selection

Abstract Binary classification models are critical in public health and medicine for diagnosis, risk stratification, and outcome prediction. Adding new variables to improve prediction is key, yet performance is commonly assessed using changes in the area under the ROC curve (ΔAUC). However, ROC curves have limitations: they provide only global evaluation, ignore class-specific effects, and may miss subtle but meaningful prediction improvements. The U-smile method offers a class-stratified alternative with graphical and quantitative tools. It evaluates prediction changes using BA (absolute), RB (relative), and I (proportional) coefficients for events and non-events separately. We compared the U-smile method with ROC curves and illustrated how the U-smile method overcomes the ROC curve's limitations. We used the Heart Disease dataset and generated random variables (normal, exponential, binomial) in balanced scenarios. Nine new variables were tested in informative and non-informative conditions. Variables were evaluated using U-smile plots and ΔAUC (DeLong's test). Statistical significance was assessed using likelihood-ratio tests (LRT) with α = 0.05. U-smile plots effectively distinguished informative from non-informative variables. ‘Smiles’ corresponded to significant LRT results (p < 0.05), while ΔAUC values were sometimes inconclusive. Two variable pairs showed nearly identical ΔAUC but different U-smile plots, revealing class-specific improvements missed by ROC analysis. One variable missed by DeLong's test was detected by U-smile. Findings were consistent in both training and test datasets. The U-smile method outperformed ROC analysis in detecting informative predictors, offering clearer, class-stratified, and more interpretable insights. Its visual format allows rapid assessment and highlights subtle but important improvements, supporting fairer, more transparent digital health tools. Key messages • The U-smile method provides class-stratified insights, outperforming ROC curves in variable selection. • Transparent methods like U-smile can advance fair and interpretable digital health tools.