Visual Saliency Prediction for Augmented Reality Videos

Augmented Reality (AR) is an emerging technology that allows users to perceive both virtual-world contents and real-world scenes simultaneously. It has numerous applications in industrial manufacturing, entertainment, gaming, education, etc. In AR environments, the visual confusion phenomenon caused by the overlay of augmented content and real backgrounds is evident, yet the understanding and research of visual saliency under the AR visual confusion condition remains limited. This paper primarily analyzes the interaction between real-world scenes and AR content, explores human visual saliency when using AR devices. First, we conduct a large-scale eye-tracking experiment based on a head-mounted AR device, and construct an AR saliency dataset containing 2160 videos, with corresponding collected eye movement data. Through qualitative analysis of the visual attention heat maps, we conclude that visual confusion significantly influences visual attention in AR video. Additionally, we quantitatively evaluate the performance of a series of classical saliency models and deep neural network saliency models on the dataset constructed in this project. For better predicting saliency in AR, we propose a general saliency prediction model, InternSal, which achieves state-of-the-art performance compared to other methods. The database and codes will be released to facilitate future research.