S483 Multi-Modal Deep Learning for Predicting Colorectal Cancer Progression and Treatment Response Using Genomic Data and Capsule Endoscopy Images

Introduction: Colorectal cancer (CRC) remains a leading cause of cancer-related deaths globally. Accurate staging, early detection of metastasis, and individualized treatment plans are crucial to improving patient outcomes. Traditional diagnostic tools, while effective, often rely on isolated datasets, such as genomic data or imaging, to predict disease progression and therapeutic responses. Recent advances in deep learning allow for the integration of multi-modal data, presenting a promising direction for enhancing cancer diagnostics and treatment decision-making. Methods: In this study, we developed a multi-modal deep learning model to predict CRC progression and treatment response by combining genomic data from the MSI/MSS dataset with capsule endoscopy images from the KYU Capsule dataset. The genomic dataset included key features such as gene expression levels and microsatellite instability (MSI) status, while capsule endoscopy images were processed using a pre-trained ResNet50 convolutional neural network (CNN) model. The fused features from both modalities were passed through a final prediction layer to classify cancer stage, predict metastasis to the small bowel, and estimate treatment response to chemotherapy. Results: Our multi-modal model demonstrated robust performance across key prediction tasks. The model achieved an 88% accuracy for classifying CRC cancer stage, with a sensitivity of 85% and specificity of 90% for early and advanced-stage differentiation. For metastasis prediction, the model achieved an AUC of 0.92, with a sensitivity of 88% and specificity of 91%. In predicting treatment response to chemotherapy, the model outperformed traditional genetic profiling by 7%, achieving an accuracy of 85% and an area under the curve of 0.87. Notably, SHAP values indicated that KRAS mutation status and MSI were key predictors for cancer stage, while capsule endoscopy findings and tumor suppressor gene expression were critical for metastasis detection. Conclusion: This research showcases the potential of integrating genomic and imaging data through deep learning to enhance CRC diagnostics. By combining capsule endoscopy images and genomic biomarkers, our model provides a more comprehensive understanding of CRC progression, metastasis, and treatment response, offering significant promise for personalized medicine in colorectal cancer management. The results highlight the potential for future applications in clinical settings, where multi-modal models can drive more accurate and timely treatment decisions.