Analyzing Crowd- and Screen-Based Exploration Behavior Compared to HMD Viewing in 360° Videos

Introduction Omnidirectional videos (ODV) have become increasingly relevant in multimedia research and applications. Most prior studies focused on viewing these videos with head-mounted displays (HMDs) in controlled lab environments, requiring complex setups and costly hardware. This limits the feasibility of large-scale ODV studies for user behaviour analysis, as not everyone owns an HMD at home. Since ODV salience reflects exploration behaviour more closely aligned with real-world viewing compared to 2D video, alternative and more scalable approaches are of high interest. Methodology Towards this end, this paper analyses to what extent exploration behaviour observed in screen-based setups (lab, out-of-the-lab, and crowdsourcing) is similar to HMD-based viewing. A framework for screen-based ODV playback and data collection was developed, using mouse movements to change the corresponding viewport. Using 20 ODV sequences, three screen-based tests were conducted: a crowd-sourced test with 45 participants, an out-of-the-lab test with 45 participants, and a lab test with 33 participants. The results were compared with an existing HMD-based lab study using a similar design. Results and conclusions For evaluation of the user behaviour, Similarity Ring Metric (SRM) scores and intentional head movement counts are used. The findings show that exploration patterns in screen-based viewing differ from those in HMD viewing, meaning the two approaches are not fully equivalent. Nevertheless, for specific applications such as general viewport prediction, screen-based data can provide a viable alternative, offering potential for improved viewport recommendations in non-immersive viewing scenarios. Furthermore, using the fixation maps from the behavioural data of all tests six saliency models have been evaluated using established evaluation metrics such as the correlation coefficient. In the HMD-based lab test, SalViT360 achieved the highest prediction accuracy (especially at higher frame aggregation levels), followed by SUM, MDS-ViTNet, and DeepGaze IIE, while UNISAL performed worst but still outperformed V-BMS360, whereas all models showed reduced accuracy in the screen-based tests. To foster reproducibility and open science, both the collected datasets and the adapted framework for screen-based testing have been made publicly available.