Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With Transformer

Remote sensing platforms are often equipped with sensors of multiple spectrums to capture the diverse reflective properties of ground areas, typically including the visible spectrum and the near infrared (NIR) spectrum. Moreover, thermal infrared (TIR) sensors capture the radiated heat of targets and are capable of all-day observation regardless of illumination conditions. By leveraging the complementary features of different spectrums, multispectral fusion techniques enhance the precision and robustness of remote sensing object detection methods. In this article, we present an object detection method for remote sensing imagery named multispectral detection transformer (multispectral DETR). The model fuses multispectral features with deformable attention and utilizes fused features for object detection. The multispectral deformable attention fusion block integrates the flexibility of dynamic weights with the principle of fusion based on local regions. Then, we propose a simple yet effective oriented object detection scheme based on angle prediction. Finally, we introduce a novel cross-teaching method between single-spectral and multispectral models, which alleviates the spectral interference issue caused by inconsistent target visibility. Experimental results demonstrate that multispectral DETR achieves state-of-the-art results on both the RGB-NIR VEDAI and the RGB-TIR DroneVehicle datasets.