Extracting and Clustering Human Visual Strategies from Video Gaze Data with Hidden Markov Models for Video-QA Performance Understanding

Human gaze patterns during tasked viewing of an action offer insights into human attention, intention recognition, and the cognitive state of the viewer. These insights have applications in many domains such as collaborative robotics, user interface design, and medical assessment. In this work, we demonstrate an association between visual strategies employed during task-driven viewing and performance in video question answering (Video-QA). To do so, we leverage eye-tracking data to analyze human visual strategies during action recognition. Specifically, we record participants' gaze as they watch demonstrations of short atomic actions, which form a basis to describe all long form actions, followed by a questionanswering step to assess their comprehension. We model the spatio-temporal dynamics of gaze behavior using Hidden Markov Models (HMMs) and cluster visual strategies for demonstrations of the same task based on visual behavior similarity. We use Binomial Generalized Linear Mixed Model (BGLMM) to investigate how this behavior relates to Video-QA performance. Our results show that participants who exhibit specific visual strategies, as well as participants who are consistent in their visual strategy across different videos, perform better on the question-answering task, while other visual strategies are associated with poor performance and can reveal misunderstandings of the questions. Specifically, more dynamic gaze behavior, with more focused fixations on a relevant object, and more relevant objects considered in the scene are all properties of a better performing visual strategy.