A Diagnostic Framework for Assessing Cross-Source Generalization in Materials Informatics: Evidence from 1231 Lightweight Aggregate Experiments Across 78 Sources

Predictive models of lightweight aggregate (LWA) properties are renowned for their excellent prediction performance within a single source of data, but their generalization power across multiple literature sources is highly questionable. Examination of 1231 observations of LWAs derived from 78 sources (2004–2026 years) and characterized by 120 attributes showed that the pooled Spearman correlation between particle density and water absorption was equal to −0.397 (N = 631; bootstrap 95% CI = −0.459 to −0.329). While 20 out of 22 observed within-source finite correlations were of the same negative sign as the pooled one, the correlations ranged [−0.987, +0.533]. The considerable variability in both magnitude and sign suggests a Simpson-type aggregation effect where the overall coefficient is masking source-level differences. In the case of the complete-case production dataset (N = 278), the Random Forest algorithm reached a mean 5-fold R2 of 0.158 but failed in LOSO generalization (mean R2 = −3.345 over nine sources). Imputation of predictors increased the sample size to 693 but degraded the LOSO performance (mean R2 = −7.634; −4.108 when restricted to the same nine sources as the complete-case analysis), demonstrating once again that predictor imputation does not improve cross-source transferability. Source level features helped random-fold generalization reach R2 ≈ 0.21 but are unable to capture unseen sources. The results presented above demonstrate how source heterogeneity, missing data and the choice of validation strategy affect the performance of the models. We propose the Cross-Source Generalization Diagnostic framework for evaluating the effects of sources, within-source relationships, performance on unseen sources, and coverage of data before the deployment of the pooled materials-informatics models.