Learning Local Features by Jointly Semantic-Guided and Task Rewards

Learning local features is a fundamental task for many computer vision applications. Existing methods often struggle to maintain robustness and accuracy in extracting local features, especially in complex environments with numerous interfering objects. Although some studies have integrated semantic information into local feature extraction networks to enhance discrimination, their effectiveness remains limited. Therefore, this paper fully considers the importance of semantic information for feature extraction and proposes a semantically enhanced local feature extraction network framework. This framework includes a local feature network, a semantic segmentation network, and a reinforcement learning framework. Semantic information is incorporated into feature heatmaps and feature descriptors to improve the accuracy of feature points. Subsequently, the local feature network is continuously optimized by a reinforcement learning algorithm based on semantic information and matching ground truth to enhance robustness, ensuring that the final local features achieve optimal performance. Extensive experiments on three publicly available datasets validate the effectiveness of the proposed local feature network.