UAMFDet: Acoustic‐Optical Fusion for Underwater Multi‐Modal Object Detection

ABSTRACT Underwater object detection serves as a crucial means for autonomous underwater vehicles (AUVs) to gain awareness of their surroundings. Currently, AUVs predominantly depend on underwater optical cameras or sonar sensing techniques to furnish vital information sources for subsequent tasks such as underwater rescue and mining exploration. However, the influence of underwater light attenuation or significant background noise often leads to the failure of either the acoustic or optical sensor. Consequently, the traditional single‐modal object detection network, which relies exclusively on either the optical or acoustic modality, struggles to adapt to the varying complexities of underwater environments. To address this challenge, this paper proposes a novel underwater acoustic‐optical fusion‐based underwater multi‐modal object detection paradigm termed UAMFDet, which fuses highly misaligned acoustic‐optical features in the spatial dimension at both the fine‐grained level and the instance level. First, we propose a multi‐modal deformable self‐aligned feature fusion module to adaptively capture feature dependencies between multi‐modal targets, and perform self‐aligned multi‐modal fine‐grained feature fusion by differential fusion. Then a multi‐modal instance‐level feature matching network is designed. It matches multi‐modal instance features by a lightweight cross‐attention mechanism and performs differential fusion to achieve instance‐level feature fusion. In addition, we establish a data set dedicated to underwater acoustic‐optical fusion object detection tasks called UAOF, and conduct a large number of experiments on the UAOF data set to verify the effectiveness of UAMFDet.