Multi-task deep learning framework for brain tumor classification and segmentation using ensemble feature extraction

Brain tumors are one of the most severe neurologic diseases, and their early and proper diagnosis is critical in developing treatment plans. MRI is often used in tumor detection; however, the manual examination is time-consuming and susceptible to human error. While deep learning algorithms have contributed to more automated brain tumor detection, there have been challenges in their implementation. For example, most methods implement classification and segmentation operations separately which increases the computational burden. In light of this challenge, this paper presents a multi-tasking deep learning architecture to detect brain tumors while incorporating both tumor segmentation and classification into one framework. Specifically, the model adopts an ensemble feature extraction technique through the use of ResNet-50, EfficientNet-B5, and convolutional neural networks (CNNs) along with attention-based modules to extract a wide range of features. A dynamic weighting process will be used to achieve effective fusion of the extracted features. In addition, classification will be the first step, and if a tumor is detected, then the U-net++ segmentation will be initiated. Experiments conducted using public MRI dataset yielded impressive results with a classification accuracy rate of 99.84%, dice coefficient of 91.22%, and IoU of 89.01%.