GazeViT: A gaze-guided hybrid attention vision transformer for cross-view matching of street-to-aerial images

• The first work to apply eye-movement attention mechanisms in the task of image cross-view matching. • Gaze information from the human brain is utilized to guide model training and learning; they are not required during testing. • The fusion of the eye-movement and self-attention mechanisms using sub-image block-level can significantly guide model to focus more on the key image regions. • The implementation of image adaptive cropping and resolution enhancement strategies effectively removes the interference of redundant information and reduces the waste of computational resources. The goal of cross-view matching between street and aerial images is to retrieve aerial-view images that correspond to a given street-view image from a database of GPS-tagged aerial images. This task relies on image cross-view matching technology, focusing on the extraction and alignment of features representing the same location in both image types. The significant differences in perspective and appearance between street-view images and aerial-view images present a challenge. Aerial-view images cover a broader area, while street-view images focus on specific locations, creating information asymmetry that complicates the matching process. To tackle these challenges, this paper proposes a gaze-guided hybrid attention Vision Transformer, which uses gaze information to guide the model to focus on and align task-related features. Furthermore, inspired by the human visual cognitive process of "focus and zoom," we develop a hybrid attention module alongside an image adaptive cropping and resolution enhancement module. The hybrid attention module utilizes gaze information to guide the model to focus on relevant regions, while the image adaptive cropping strategy uses gaze information to guide the model to eliminate irrelevant regions. Techniques for improving image resolution allow for the magnification of important regions, thereby aiding the extraction of fine-grained features. We evaluate the model's performance on benchmark datasets and conduct ablation study experiments to assess the contributions of each module. Experimental results show that the method achieves a top-1 accuracy of 75.56% on the CVACT dataset, representing state-of-the-art performance. This study provides valuable insights into incorporating human experience into computational models, particularly through gaze-guided learning of visual task networks to enhance model performance.