Deep Fusion of Visible and Near Infrared Images for Registration and Defogging Using Cross Modal Transformer

According to Rayleigh scattering, the density of fog is closely related to the distance of the target from the visible light camera. However, the near infrared (NIR) spectral band is robust to the interference problem but has a single channel without color. In this paper, we propose deep fusion of visible (VIS) and near infrared (NIR) images for registration and defogging using cross modal transformer. The proposed fusion network has two main features: 1) Cross modal transformer deals with the channel and spatial redundancy between two source images while retaining their complementary information; 2) Since the blue channel in VIS image can represent the amount of fog in the scene according to Rayleigh scattering, we normalize it to obtain a fog weight map. To synthesize training data, the distance between object and camera is required, thus we use depth estimation to obtain it. Then, we assign different amounts of fog to each image to consider various weather conditions. First, we extract multi-scale features from the source images. Second, we use the cross modal transformer to reflect cross-domain interaction between the source images in spatial and channel domains. Third, we generate the fusion weight for the source images according to the fog weight map and reconstruct a fused result based on it. Various experiments on the widely-used benchmark and real-world datasets show that the proposed method generates natural-looking defogging results with clear textures in distant regions while keeping the original tone in near regions. Moreover, the proposed method outperforms state-of-the-art fusion methods in term of both visual quality and quantitative measurements.