Development and validation of a multimodal model based on histopathological and computed tomography imaging for predicting survival outcomes in stage III colorectal cancer

Abstract Stage III colorectal cancer (CRC) patients exhibit substantial variability in survival outcomes despite standardized Tumor–Node–Metastasis (TNM) staging and oxaliplatin‐based adjuvant chemotherapy. Current prognostic models often rely on single‐modality data, limiting their predictive accuracy and clinical utility. We developed a novel Multimodal Prognostic Index (MMPI) that integrates histopathological features from hematoxylin and eosin–stained whole‐slide images and radiomic features from preoperative computed tomography scans. This retrospective, multicenter study included 253 stage III CRC patients from three institutions, all of whom received oxaliplatin‐based adjuvant chemotherapy. Prognostic predictions were generated using a robust, risk‐aware joint and individual representation learning algorithm (Robust risk‐Aware Joint and Individual [RAJI]). MMPI was trained and tested on internal cohorts and validated externally. Feature contributions were interpreted using SHapley Additive exPlanations. MMPI consistently outperformed unimodal models based solely on pathomics or radiomics in predicting overall survival (concordance index: training set, 0.716 vs. 0.596–0.620; testing set, 0.672 vs. 0.594–0.642; external validation set, 0.632 vs. 0.580–0.605) and recurrence‐free survival (C‐index: training set, 0.713 vs. 0.609–0.656; testing set, 0.735 vs. 0.611–0.639). It remained an independent prognostic factor after multivariate adjustment and improved risk stratification within TNM‐defined subgroups. Furthermore, MMPI effectively predicted chemotherapy efficacy, particularly distinguishing high‐risk patients less responsive to 3‐ or 6‐month XELOX and FOLFOX regimens. This study is the first to integrate radiological and pathological imaging for prognostic assessment in stage III CRC. It offers a non‐invasive, interpretable tool to improve survival prediction and guide individualized chemotherapy decisions.