PathoHR: Using Clinical Data and Pathology Images to Predict Breast Cancer Survival

Because of patient-specific variable complexity and tumor heterogeneity, predicting survival in breast cancer is one of the most challenging computational pathology tasks. Prediction accuracy has been improved by using features from high-resolution pathological images in recent studies like PathoHR, though these were applied from only image data and may not be able to take advantage of valuable clinical and genetic information. To better predict survival, we introduce here the first multi-modal learning-based method, PathoHR-M, to integrate high-resolution pathology images with further clinical and genetic data. We use an MLP to learn to represent further non-image data and a plug-and-play Vision Transformer (ViT) to extract patch-wise image features. The two modalities are subsequently combined through a fusion module before prediction. Experimental results on the TCGA-BRCA data set indicate PathoHR-M to outperform substantially the baseline PathoHR model by gigantic margins for all the AUC(0.96), F1-score(0.94), and accuracy(0.95) metrics, paving the way towards more sophisticated survival prediction models in computational pathology.