Image characteristics, not task, influence interobserver consistency

Interobserver consistency (IOC) refers to the degree of similarity in the gaze patterns of different observers when viewing the same image. Here, we investigated the role of image characteristics and viewing task on IOC. We recorded the eye movements of 80 participants while they viewed 192 images of scenes from the ADE20K dataset (Zhou et al., 2017). To vary task demand, participants either freely viewed the images or answered a content-related question following each image. We also considered how two image characteristics – visual clutter and semantic complexity – might impact IOC. We calculated visual clutter using the Feature Congestion measure (Rosenholtz et al., 2005). Additionally, to calculate semantic complexity, we utilised the object labels available for the ADE20K images. First, we used a distributional semantic model (GloVe; Pennington et al., 2014) to compute the pairwise semantic dissimilarity between the object labels in each image. Then, we averaged these values to derive an overall 'semantic complexity' score per image, where images with greater dissimilarity among their objects were considered more complex. Interobserver consistency was assessed by how well the fixation heatmap of all observers, excluding one, predicted the fixation heatmap of the remaining observer, using Pearson correlation coefficient. We used a linear mixed-effects model to examine the relationship between visual clutter, semantic complexity and task condition on IOC. Our results revealed an effect of visual clutter (p < .001), with IOC decreasing as clutter increased. In contrast, semantic complexity showed a positive association with IOC (p = .033), suggesting increased consistency with greater semantic complexity. Intriguingly, we found no effect of task on IOC. Taken together, our study highlights the nuanced relationship between image characteristics and IOC. Furthermore, the surprising lack of task effect indicates that, in our study, image characteristics play a more prominent role in influencing interobserver consistency.