MMAformer: Multiscale Modality-Aware Transformer for Medical Image Segmentation

The segmentation of medical images, particularly for brain tumors, is essential for clinical diagnosis and treatment planning. In this study, we proposed MMAformer, a Multiscale Modality-Aware Transformer model, which is designed for segmenting brain tumors by utilizing multimodality magnetic resonance imaging (MRI). Complementary information between different sequences helps the model delineate tumor boundaries and distinguish different tumor tissues. To enable the model to acquire the complementary information between related sequences, MMAformer employs a multistage encoder, which uses a cross-modal downsampling (CMD) block for learning and integrating the complementary information between sequences at different scales. In order to effectively fuse the various information extracted by the encoder, the Multimodal Gated Aggregation (MGA) block combines the dual attention mechanism and multi-gated clustering to effectively fuse the spatial, channel, and modal features of different MRI sequences. In the comparison experiments on the BraTS2020 and BraTS2021 datasets, the average Dice score of MMAformer reached 86.3% and 91.53%, respectively, indicating that MMAformer surpasses the current state-of-the-art approaches. MMAformer’s innovative architecture, which effectively captures and integrates multimodal information at various scales, offers a promising solution for tackling complex medical image segmentation challenges.