Multiscale Reliability-Guided Dual Attention Fusion Network for Multimodal Object Detection

Abstract Multimodal images (RGB and thermal images) can provide complementary information, making object detection more reliable and robust. However, most existing methods suffer from two issues: 1) interference from low-quality modality information in extreme environments (e.g., darkness, overexposure, and bad weather); 2) insufficient utilization of intra- and inter-modality information. To overcome these two issues, we propose a multiscale reliability-guided dual attention fusion network (MRDAN). In MRDAN, we first propose a multiscale reliability-guided module (MRGM). MRGM uses multiple parallel dilated convolution branches to calculate the reliability scores of RGB and thermal features at each level, and then uses these scores to enhance high-quality modality features and suppress low-quality modality ones. Then, we design a dual attention fusion module (DAFM). DAFM adopts parallel global and local attention units to capture long-range inter-modality contextual information and specific intra-modality information, respectively, and aggregates these two types of information for further performance improvement of object detection. Results on two benchmark datasets (FLIR and LLVIP) show that MRDAN achieves better overall performance than state-of-the-art multimodal object detection methods.