A Methodological Framework for Studying Online Language–Vision Interplay in Human–Robot Interaction

Abstract Investigating the real-time interplay between language and vision has traditionally involved a trade-off between experimental control and interactive setups. To address this challenge, we present a methodological framework that embeds the visual world paradigm (VWP) within a continuous, goal-directed human–robot interaction (HRI) task. By augmenting VWP in an HRI setup to produce standardized yet responsive verbal and non-verbal behaviours, the framework makes it possible to selectively examine comprehension, production, actions and their relation to visual attention in real-time. We demonstrate the utility of the framework through a proof-of-concept manipulating action structure, specifically path and manner, as well as other-initiated repair formats, namely open and restricted requests. The findings show that the setup reliably captures time-locked referential gaze dynamics following comprehension and preceding production processes. They further indicate that differences in action history systematically influence speakers’ pre-verbal cognitive processes and their subsequent linguistic choices. These findings suggest that HRI can provide a useful methodological middle ground to investigate language processes online in a tightly controlled setup without disrupting the interaction. More broadly, the framework offers a test bed for examining whether established psycholinguistic findings extend to HRI setups. At the same time, it provides HRI researchers with an empirical basis for developing adaptive agents informed by online language-vision interaction theories.