Development and cross-cohort evaluation of a multimodal deep-learning model for ER, PR and HER2 status classification in breast cancer

Background. Estrogen receptor (ER), progesterone receptor (PR) and HER2 status guide systemic treatment in breast cancer. Multimodal models are rarely benchmarked against key biomarker molecular baselines or evaluated for cross-cohort calibration. We assessed incremental value, dependence on label-proximal HER2 transcripts, and transport of discrimination and calibration. Methods. A model combining whole-slide images, transcriptomes, age and stage was developed in TCGA-BRCA (1051 patients; ER/PR/HER2 analysis sets: 1001/998/1006) and evaluated in CPTAC-BRCA (120 patients; 118/113/95). CPTAC had been observed during earlier development and included 13 HER2-positive cases. Analyses included paired model comparisons, transcript removal, alternative loss functions and recalibration. Internal discrimination used pooled out-of-fold predictions; external discrimination used fold-mean estimates, with ensemble calibration reported separately. Internal calibration and macro-AUC uncertainty were interpreted as exploratory. Results. Internal AUCs were 0.943 for ER, 0.901 for PR, and 0.860 for HER2; external AUCs were 0.952, 0.937, and 0.918, respectively. External macro-AUC was 0.936 for the multimodal model and 0.957 for logistic regression using ESR1, PGR, and ERBB2 (difference -0.021). Removal of the extended 29-transcript 17q12-q21 region changed HER2 AUC by -0.035 [-0.058 to -0.013] (95% CI) internally and by -0.042 [-0.089 to -0.007] externally relative to the baseline multimodal model. For external 15-model ensemble predictions, recalibrated HER2 expected calibration error (ECE) was 0.236, exceeding the ECE reference ceiling of 0.125, while calibration-in-the-large was -1.657 [-2.373 to -1.023] (profile-likelihood 95% CI), indicating systematic overprediction. ER and PR also showed overprediction. In paired fold-mean comparisons, HER2 ECE and Brier score favoured the three-transcript model (ESR1, PGR, and ERBB2). Conclusions. A multimodal deep-learning model achieved high discrimination across cohorts but did not demonstrate clear incremental value over a parsimonious three-transcript baseline, while cross-cohort calibration remained a substantial challenge. These findings highlight the importance of evaluating incremental value, calibration, and clinical utility alongside discrimination when assessing AI-based prediction models for medical use.