MMGFFN: A Multimodal Multi-Granularity Feature Fusion Network for Cancer Survival Prediction

Survival prediction is a challenging ordinal regression task. Despite significant advancements in cancer survival prediction methods that integrate pathology and genomics, existing approaches still face challenges: they overlook the pyramid organizational structure characteristics of whole slide image (WSI) and the issue of redundant information in multimodal feature fusion. To address these challenges, we propose a novel multimodal multi-granularity feature fusion network (MMGFFN) for cancer survival prediction, aiming to fully leverage the multi-granularity hierarchical information in pathology images and achieve effective multimodal information fusion. Specifically, we first categorize WSI at different magnifications into different granularity levels, extracting coarse-grained features of tissue morphology at low magnification and fine-grained features at the cellular level at high magnification to fully utilize the information in the pathology images. We then construct a multimodal hierarchical attention module to facilitate the interaction of multimodal multi-granularity features. Finally, we employ a rough attention to dynamically weigh the multimodal and multi-granularity features, reducing redundancy and enhancing the performance of survival prediction. We conducted extensive experiments on three cancer datasets, and the results demonstrate that our method achieves significant improvements in cancer survival prediction.