Maritime ship target detection based on visible and infrared modal image fusion

Abstract The deep learning based maritime ship target detection is a key technology in fields such as ship navigation, water surface security, and military early warning. In view of the inherent limitations of maritime vessel object detection in single modality, a novel YOLO for maritime vessel object detection according to the visible and infrared modality images fusion (VIMF-YOLO) is built. The VIMF-YOLO is improved from YOLO v8 and which can effectively extract and aggregate the features of different modal ship target images. Additionally, it employs dual-modal fusion module (DMFM) to adaptively weight and fuse the different modalities features of vessel images in visible and infrared, thereby fully leveraging the complementary superiority of these modalities. To better acquire channel and positional information of different modal features, efficient multi-scale attention (EMA) is introduced into DMFM and VIMF-YOLO networks to improve the representation ability of different modal features. In addition, a paired image dataset for visible and infrared maritime ship images is built, and a large number detection test experiments for VIMF-YOLO is conducted on this basis. The experimental results prove that, matched with current SOTA ship target detection algorithms, the dual-modal fusion detection algorithm VIMF-YOLO exhibits superior detection accuracy.