Transforming Tabular Data for Multi-Modality: Enhancing Breast Cancer Metastasis Prediction Through Data Conversion

Breast cancer metastasis prediction plays a key role in clinical decision-making and secondary analysis. Traditionally, metastasis classification models have been developed using structured tabular clinical data, but these approaches may result in data loss and lack of contextual information. A multi-modal approach is presented in this article for predicting breast cancer metastasis by converting structured clinical data into unstructured text, which provides more contextual information, and then converting that text into histopathology images. For text classification, features were extracted and fine-tuned. Using Logistic Regression and XGBoost classifiers, these extracted features exhibited enhanced performance. The accuracy of the metastasis detection was further enhanced by fine-tuning. A pre-trained diffusion model was used to generate histopathology images based on the same clinical data to address the multimodality gap. The classification of the features extracted from these images, using pre-trained vision models like VGG-16 and ViT, provided similar results to traditional tabular predictions. A multi-modal early fusion approach was then created by combining vision-derived features with text-based features from the BERT. Using unstructured text and histopathology images can effectively address multi-modal data limitations, providing a promising alternative for future research and providing a context-rich approach to breast cancer metastasis prediction.