Selective Masked Auto-Encoders Are Self-Supervised Visual Key-Points Learners

This paper presents a novel approach in key-point learning by integrating Masked Auto-Encoders (MAE) for self-supervised learning of local key-point representations. Different from traditional contrastive learning methods, our approach specifically focuses on reconstructing masked parts of images so as to enhance feature comprehension and model generalization while reducing computational requirements. We introduce a selective masking strategy that synergizes the key-point detection with image reconstruction tasks, which leverages the shared Transformer blocks to facilitate joint optimization and improve key-point localization accuracy. Our methodology employs a fully self-supervised learning paradigm on natural images such that the need is avoided for datasets with additional annotations like SfM or camera pose information. Experimental results show that our approach achieves comparable or superior performance to existing contrastive learning-based methods on standard image-matching benchmarks and demonstrates the remarkable generalization capabilities in various computer vision tasks, including visual odometry.