Segmentation of Brain Tumors Using a Multi-Modal Segment Anything Model (MSAM) with Missing Modality Adaptation

This paper presents a novel multi-modal segment anything model (MSAM) for glioma tumor segmentation using structural MRI images and diffusion tensor imaging data. We designed an effective multimodal feature fusion block to effectively integrate features from different modalities of data, thereby improving the accuracy of brain tumor segmentation. We have designed an effective missing modality training method to address the issue of missing modalities in actual clinical scenarios. To evaluate the effectiveness of MSAM, a series of experiments were conducted comparing its performance with U-Net across various modality combinations. The results demonstrate that MSAM consistently outperforms U-Net in terms of both Dice Similarity Coefficient and 95% Hausdorff Distance, particularly when structural modality data are used alone. Through feature visualization and the use of missing modality training, we show that MSAM can effectively adapt to missing data, providing robust segmentation even when key modalities are absent. Additionally, segmentation accuracy is influenced by tumor region size, with smaller regions presenting more challenges. These findings underscore the potential of MSAM in clinical applications where incomplete data or varying tumor sizes are prevalent.