Feature leakage inflates machine-learning estimates of antiretroviral non-adherence risk in children living with HIV: a tiered sensitivity analysis of the EARTH cohort

Abstract Objective A companion analysis of the EARTH cohort developed machine-learning models to identify suboptimal antiretroviral therapy adherence in children living with HIV after excluding adherence-process fields that directly contributed to the outcome definition. This Research Note quantifies how retaining those fields changes apparent model performance. Using a separate sensitivity-analysis extract from the same cohort, we compared three nested predictor tiers across 590 visits from 208 children: leakage-naive, loosely controlled, and strictly controlled, using fourteen classifiers and grouped five-fold cross-validation. Results Including outcome-defining adherence-process fields markedly inflated discrimination across all fourteen classifiers, increasing AUROC by 0.24–0.41 and AUPRC by 0.33–0.68; six classifiers reached AUROC 1.000. After removing these fields, CatBoost achieved AUROC 0.735. Excluding an additional ART treatment-interruption variable, which dominated permutation importance despite not formally defining the outcome, reduced CatBoost AUROC to 0.680 and AUPRC to 0.407, closely matching the companion analysis. The findings show that adherence-process variables can create both direct and near-label leakage, producing deceptively strong performance. Tiered leakage auditing should therefore accompany machine-learning studies that predict adherence from routinely collected process data, particularly when predictors overlap conceptually with outcome construction before interpreting apparent performance as clinically useful predictive signal.