Towards advanced brain tumor segmentation: a novel hybrid architecture integrating UNet, FCN, and YOLO models on the newly introduced BTS-DS 2024 dataset

Abstract To advance the field of brain tumor segmentation, we introduce the Brain Tumor Segmentation Dataset 2024 (BTS-DS 2024), a publicly available dataset comprising 3956 MRI images classified into 14 tumor types. The dataset includes T1-weighted, contrast-enhanced T1 (T1C+), and T2 MRI modalities, providing a comprehensive foundation for segmentation and classification tasks. Using the BTS-DS 2024 dataset, we developed a comprehensive suite of segmentation models, including UNet, ResUNet, Fully Convolutional Networks (FCN), VGG16, and their hybrid variations. In addition, we integrated cutting-edge YOLO architectures—YOLOv8, YOLOv9, and YOLOv11—to assess their effectiveness in brain tumor detection. The experimental results underscore the outstanding performance of YOLOv9e, which achieved the highest $$F_1$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msub> <mml:mi>F</mml:mi> <mml:mn>1</mml:mn> </mml:msub> </mml:math> score of 92.2538% and an Intersection over Union (IoU) of 85.6214%, demonstrating a strong balance between precision and recall. Moreover, YOLOv11x achieved the highest mean Average Precision (mAP50-95) value of 74.3835%, showcasing superior accuracy across multiple detection thresholds. The originality of this study is exemplified by the introduction of the novel BTS-DS 2024 dataset and the development of models that establish new best-performing scores for $$F_1$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msub> <mml:mi>F</mml:mi> <mml:mn>1</mml:mn> </mml:msub> </mml:math> , IoU, and mAP50-95 in the literature. These contributions significantly advance the state-of-the-art in brain tumor segmentation, establishing a robust foundation for future research in this domain.