A multimodal deep learning approach for the prognostic stratification of patients with colorectal cancer (CRC).

287 Background: Traditional prognostic systems like the AJCC TNM staging for colorectal cancer (CRC) often fall short in predicting long-term patient outcomes. These systems often rely on limited pathologic features, leading to generalized approaches to treatment despite diverse tumor heterogeneity, such as the complex interaction between the tumor-host microenvironment. There is therefore a pressing need for more accurate, scalable tools to enhance decision-making and predict outcomes for patients with CRC. Methods: Tumor sections from 191 CRC patients were comprehensively stained for CD8 to examine CD8-positive tumor-infiltrating lymphocytes (TILs). To account for tumor heterogeneity, multiple areas of the tumor were evaluated to include intertumoral and peritumoral areas. Slides were digitized and quantitatively analyzed using a cloud-based platform (SpatialX Diagnostics, Inc. USA). Additionally, clinicopathologic features including patient demographics, 8th edition AJCC staging, lymphovascular invasion (LVI), perineural invasion (PNI), mismatch repair (MMR) protein status, tumor location, presence/development of distant metastases, and patient follow-up were collected. Merging CD8 AI findings with clinicopathologic features, a complete data set was available for 175 patients. Cases were randomly assigned into 70% training and 30% validation. In the training set, unsupervised artificial intelligence (AI) models integrated clinicopathologic data and CD8-positive TIL counts using vision transformer models. The final model combined spatially resolved pathologic features with clinical variables to statistically predict overall survival (OS) and distant metastasis risk (DMR). Results: In the validation cohort, the model stratified patients into two groups based on a deep learning-derived risk score, with statistically significant differences between low-risk and high-risk groups for OS (p<0.02) and DMR prediction (p<0.0001), For OS prediction, the model selected tumor site, PNI, MMR status, epithelial and stromal CD8-positive TILs, and 5 deep vision features as the top 10 clinically significant features. For DMR prediction, the age, T- and N-stage, PNI, and 6 deep vision-derived features as key predictors. Conclusions: As captured by vision transformers, the distribution and spatial arrangement of CD8-positive TILs are critical factors in patient stratification for OS and DMR. AI-driven models like the one herein offer the potential for more personalized management strategies, advancing CRC management and improving patient outcomes beyond traditional prognostic staging systems and parameters.