CTNet: Multimodal Remote Sensing Image Keypoint Detection and Description for CNN and Transformer Architectures

Keypoint detection and description from multisensor or multimodal images are fundamental to image registration and its downstream tasks. However, the nonlinear radiometric differences, illumination variations, and geometric distortions between multimodal remote sensing images pose significant challenges. To address these issues, this paper proposes a weakly supervised multimodal keypoint detection and description network (CTNet), which extracts robust and repeatable feature descriptors at a low cost without requiring densely labeled annotations or extensive pretraining. In terms of network design, CTNet effectively combines convolutional neural network (CNN) and Transformer architectures by introducing a multimodal global and local information interaction (MGLI) module. Additionally, a lightweight keypoint detector is designed to efficiently detect keypoints by evaluating pixel saliency within neighborhoods and incorporating their depth maxima. For model optimization, a novel loss function, multiple pair weighted loss, is introduced. This loss function samples and weights positive and negative pairs of multimodal features, effectively capturing the similarity relationships among samples to learn a robust feature embedding space. Finally, CTNet is evaluated on both public and self-collected multimodal VIS-SAR and VIS-IR image datasets and compared with state-of-the-art keypoint detection and description models. Experimental results demonstrate that CTNet achieves superior matching accuracy and robustness in multimodal image matching tasks, outperforming existing methods.