Online Adaptive Keypoint Extraction for Visual Odometry Across Different Scenes

Visual odometry needs to be robust against various environmental changes. Although Deep learning (DL) based methods can bring more robust features to visual odometry (VO) than traditional methods, the gap between training and test dataset restricts the performance of DL-based methods when encountering novel scenes. Additionally, due to non-differentiability of visual odometry optimization process, the scene independent geometric constraints of images is hardly used in DL-based methods, which further decreases the accuracy and generalization of odometry. In this letter, we propose a reinforcement learning based framework incorporating the geometric constraints to address the challenge of non-differentiability. Unlike conventional DL-based methods, our VO estimates the pose through traditional direct odometry. To facilitate online training, a weighted direct graph structure is utilized to efficiently organize and simplify local images. This online training scheme enables the keypoint extraction policy to adapt dynamically to the current scene of odometry with less training data. Evaluations against the state-of-the-art visual odometry were performed on the KITTI Odometry, EuRoC MAV and Tum RGBD dataset. The results demonstrate that our method outperform popular approaches in terms of tracking accuracy and generalization ability.