Hybrid of Handcrafted Features and Learnable Features Using Multimodal Fusion Network for Brain Tumor Segmentation

ABSTRACT Diagnosing a brain tumor manually with the magnetic resonance imaging (MRI) in a clinical setting is a difficult task due to its large volume, presence of artifacts, and associated manual error. While existing Unet models and their variants have shown promise for medical image segmentation, these models often struggle to capture global contexts. On the other hand, transformer‐based architectures involve large learnable parameters and need huge computational resources, which restrict its deployment in real clinical settings with a resource‐constrained environment. Most of the existing multimodal deep learning (DL) models utilize either input layer fusion or output layer fusion. Hence, these models are unable to fuse relevant and complementary information from multiple modalities to provide effective segmentation. Few works have investigated layer‐level fusion, which has been found to be a better strategy for capturing relevant and complementary information from multiple modalities. To address these limitations, we propose HHL‐MFNet, a lightweight and cost‐effective framework for brain tumor segmentation. The proposed method utilizes layer‐level fusion along with a hybrid feature extraction strategy to combine a handcrafted Discrete Wavelet Transform (DWT) to explicitly capture high‐frequency edge details with a learnable encoder–decoder architecture. Multimodal fusion (MMF) module and the cross‐modal integration module (CIM) are used to learn a representation to preserve modality‐specific relevant details and complementary information. Evaluation on BraTS 2020 and BraTS 2021, two publicly available medical image datasets, demonstrates that the proposed HHL‐MFNet achieves superior performance in terms of dice score, HDF95, and sensitivity in comparison with eight existing models. Additionally, the HHL‐MFNet has significantly fewer learnable parameters compared to the one model whose segmentation performance is slightly better.