Prototype-Driven and Multiexpert Integrated Multimodal MR Brain Tumor Image Segmentation

For multimodal magnetic resonance (MR) brain tumor segmentation, current methods usually directly extract the discriminative features from input images for the tumor subregion category. However, information aliasing caused by the mutual inclusion of tumor subregions is often ignored. Moreover, existing methods usually do not take tailored efforts to highlight the single tumor subregion features. To this end, a multimodal MR brain tumor segmentation method with tumor prototype-driven and multiexpert integration is proposed. It could highlight the features of each tumor subregion under the guidance of tumor prototypes. Specifically, to obtain the prototypes with complete information, we propose a mutual transmission mechanism to transfer different modal features to each other. Furthermore, we devise a prototype-driven feature representation and fusion (PFRF) method, which implants the learned prototypes into tumor features and generates corresponding activation maps. With the activation maps, the subregion features consistent with the prototype category can be highlighted. A key information enhancement and fusion strategy with multiexpert integration is designed to further improve the segmentation performance. The strategy can integrate the features from different layers of the extra feature extraction network and the features highlighted by the prototypes. Experimental results on three competition datasets prove the superiority of the proposed method. The source code can be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Linzy0227/PDMINet</uri>.