Predicting Continuous Cognitive Decline: The Generalizability of a Multimodal Machine Learning Approach Including Structural MRI and Non-Brain Data

Aging is often accompanied by cognitive decline, but the extent, timing, and severity of this process is subject to large inter-individual variability. Predicting cognitive decline along a continuum, encompassing healthy age-related decline, mild cognitive impairment, and dementia, allows for more precise individual-level predictions. Previous work has demonstrated that machine learning (ML) models combining risk factors, clinical, neuropsychological, and structural magnetic resonance imaging (MRI) data can predict continuous cognitive decline. However, generalizability across independent datasets has rarely been evaluated. This study aimed to replicate previous findings using an independent dataset and to assess the models' generalizability across acquisition sites. Multi-target random forest regression models were employed to predict annual rates of decline of the Clinical Dementia Rating Scale Sum of Boxes (CDR-SOB) and the Mini-Mental State Examination (MMSE). Results showed that adding structural MRI data to non-brain data enhanced model performance in the large-scale novel ADNI (<i>N</i> = 1237) cohort, as previously attested for OASIS-3 (<i>N</i> = 662). Further, detectable reductions in performance were measured when models were tested across datasets compared to within datasets. This performance degradation was not explained by distributional shifts of the target variables. Additionally, models trained on the most important features of the respective training set achieved nearly identical performance as those trained on all features when tested externally, suggesting high redundancy among predictors. In summary, multimodal ML models predicting continuous cognitive decline generalize partially to unseen cohorts, with statistically significant performance degradation, whereas unimodal models trained on structural MRI features do not.