Masked Modality Complementary Modeling for Brain Tumor Segmentation

Self-supervised pre-training techniques based on image reconstruction have achieved substantial success in medical image analysis, allowing for the transferability of pre-trained model weights to various downstream tasks for further fine-tuning. However, current pre-training methods primarily target single-modal medical images, like CT scans, scarcely considering the multi-modal images like multi-modal brain MRI. Yet, the latter is especially pivotal for accurate tumor segmentation, given that each modality provides unique insights into the tumor’s characteristics. In this study, we introduce a self-supervised pre-training approach tailored for multi-modal brain MRI, equipped with Masked Modality Complementary Modeling (MMCM). Specifically, the proposed method involves masking a designated portion of each modality to ensure the visible parts are distinct and complementary. We assume that through the process of reconstructing such images, the model not only learns general anatomical and modality-specific characteristics but also gains insights into bridging and mapping across various modalities. Results of downstream experiments validate that our method outperforms state-of-the-art self-supervised learning methods in the tumor segmentation accuracy on the BraTS 2021 dataset. Additionally, in downstream tasks in two common scenarios: the small sample size and the single available modality, our method substantially improves the performance over the baseline model trained from scratch. The model and code are available at https://github.com/liangjianwen01/MMCM.