Co-attention-based Multi-modal Feature Alignment for Cancer Analysis

Cancer incidence and mortality rates are increasing rapidly, making accurate survival prediction for cancer patients is a critical issue in cancer research. With advancements in digital pathology and genome sequencing technologies, the integration of pathology images and omics data has become the focus of cancer diagnosis and survival prediction studies. Previous studies have insufficiently explored the optimization and calibration of features between omics data and pathological image characteristics, which affects the performance of models in patient survival prediction to tackle these challenges, we developed a co-attention-based multi-modal feature reconstruction and alignment (CMFRA) framework that aims to uncover the connections between different data modalities and efficiently optimize and align modal features. We construct feature extractors for both pathology and omics that utilize the attention mechanism to identify potential links between omics features and the cellular morphology of pathology images. Additionally, feature decoders are utilized to reconstruct the multi-modal fusion features, facilitating the optimization of unimodal features. Our experimental results demonstrate that this modality alignment method not only improves the accuracy of the model's survival prediction, but also offers the way for personalized medicine and precision therapy.