A fusion label for keypoint matching on multi-spectral images

Keypoint matching on multi-spectral images refers to identifying and establishing associations for corresponding objects or locations from two or more images of different spectral bands. Rigorous keypoint labels of real images are difficult to obtain by human annotators because of the less strict definition of keypoint. Supervised methods typically use the detection results of classical methods or pre-trained models as keypoint labels to train the learning-based detector. However, the non-linear appearance differences in multi-spectral images lead to the fact that keypoint labels of a scene are not exactly the same, which may limit the detector’s ability to detect keypoints consistently. To relieve this issue, this paper presents a method to construct fusion label for training keypoint detector on multi-spectral images. The fusion label combines keypoint information of original labels of different spectral bands. Specifically, we aggregate all points in original labels of multi-spectral images and remove the points that only exist in labels of a few spectral bands. And the fusion label is composed of the preserved keypoints that are more universal in multi-spectral images. To train the detector, we use the same fusion label for multi-spectral images of a scene, thereby fostering the consistent keypoint detection across different spectral bands. Keypoint detection and matching experiments on multi-spectral images show that using fusion label to train the detector increases the repeatability by 6.0% on average and increases the matching score