Validation-Dependent Accuracy and Limited Zero-Shot Transfer: A Cross-Database Audit of Lipid Nanoparticle Performance Prediction

Background/Objectives: Machine learning models are increasingly used to prioritize lipid nanoparticle (LNP) formulations, but apparent accuracy may depend strongly on dataset heterogeneity and validation design. This study audited two public LNP resources to determine how database consistency, source study signatures, chemical similarity, and experimental-domain shift affect predictive transportability. Methods: LNPDB and LNP Atlas were harmonized using a provenance-preserving canonical schema, standardized molecular structures, and conservative cross-database formulation matching. Source publication identity was predicted after direct bibliographic identifiers were removed. Biological-performance modeling used 17,420 study-normalized gene expression/luminescence records from 37 LNPDB studies and fixed LightGBM models evaluated under random-row, formulation-held-out, ionizable-lipid-connectivity-held-out, structural-similarity-cluster-held-out, study-held-out, and temporal validation. Results: Only 29 of 217 Atlas formulation clusters from shared publications were confirmed as complete-formulation matches. Source publication was identified with 99.94% accuracy in LNPDB, while experimental context alone achieved 99.05% accuracy in LNP Atlas. Random-row validation yielded R2 = 0.317 and ROC-AUC = 0.815. Performance decreased under connectivity holdout (R2 = 0.239; ROC-AUC = 0.761) and structural-similarity holdout (R2 = 0.035; ROC-AUC = 0.654) and was poor for unseen studies (R2 = 22120.070; ROC-AUC = 0.531) and later studies (R2 = −0.217; ROC-AUC = 0.528). Test-independent binary endpoint sensitivity reproduced the same hierarchy. Conclusions: Public LNP datasets contain strong source study signatures and substantial chemical and experimental heterogeneity. In the evaluated LightGBM and Ridge baseline models, predictive performance was strongly dependent on validation design, with limited zero-shot generalization to unseen publications and later experimental domains. These findings do not establish that the same degree of performance deterioration would occur with pretrained, graph-based, or domain-adapted LNP models. Structure-aware, publication-grouped, and temporal validation should therefore accompany claims of predictive generalization beyond the training domain.