Explainable machine learning for predicting longitudinal dementia status: Establishing a leakage-free benchmark

Dementia research often suffers from methodological pitfalls such as label-information and subject-information leakages. Leveraging the longitudinal OASIS-2 cohort, this study identifies and addresses three critical gaps in prior research: (i) inclusion of Clinical Dementia Rating (CDR) in predictor sets, risking target leakage and inflated performance estimates; (ii) inappropriate splitting strategies for longitudinal data, risking information leakage; and (iii) inadequate incorporation of temporal dynamics in classification models. To address these, CDR was excluded from input features to ensure that model performance reflects genuine predictive signals from demographic and neuroimaging data. A group-aware data-splitting strategy was implemented to maintain longitudinal data integrity and prevent leakage across training and test sets. Several engineered temporal features were also introduced to evaluate their impact on classification performance. To develop classifiers, a kernel-based classifier (support vector classifier, SVC), boosting ensembles (CatBoost, LightGBM), and bagging ensembles (Random Forest, Extra Trees), were employed. Furthermore, Explainable AI (XAI) techniques, including permutation importance and SHapley Additive exPlanations (SHAP), were utilized to interpret predictions of the best models and identify feature contributions. SVC and LightGBM (LGBM), trained on a combined feature set of original and engineered features, outperformed others. SVC achieved higher precision (73.3%) and discriminative power (ROC AUC 89.1%), while LightGBM provided better recall (69.7%) and accuracy (69.7%). XAI analyses revealed that (i) Mini-Mental State Examination and atlas-scaling-factor dynamics dominated SVC decisions, while (ii) education, age, and estimated total intracranial volume were critical for LGBM predictions. A sex-based analysis showed that both SVC and LGBM models performed better among females, which may indicate potential biases in the models. Collectively, this study establishes a transparent benchmark for leakage-free dementia status classification, demonstrating that realistic predictive performance is markedly lower once leakage is eliminated. This benchmark, together with interpretable analyses, provides a reference point for developing more robust longitudinal dementia models.