Building and coordinating multimodal interaction with gaze across two communicative situations

Abstract During face-to-face interactions, gaze plays a fundamental role in shaping the status of co-participants. A shift in gaze direction can signal a form of engagement or joint attention if it is oriented towards the interlocutor, or by contrast a form of disengagement if it is averted. Gaze also contributes to the shaping and complex development of multimodal interaction, as it enables speakers to simultaneously maintain several viewpoints who constantly align to each other’s positionings within the activity in progress. The present study focuses on the role of gaze direction and its coordination with other visible body behavior (i.e. hand gesture) found in interaction, more specifically in university settings. The analyses are carried out on the DisReg Corpus (Kosmala 2020) which comprises video recordings of French undergraduate students in two distinct situations: during individual oral presentations in class presented in front of the instructor and the students from the class, and during semi-spontaneous informal conversations between two students. The aim of this study is to illustrate the importance of gaze for building multimodal discourse across different communicative situations, combining quantitative and qualitative analyses through detailed annotations of gaze and its coordination with gestural actions. Results show that during class presentations, students look extensively towards their notes but pay very little attention to their audience, while during conversations, speakers mobilize several visual resources they have at their disposal, including gaze, to check visual attention, signal shifts in participation and viewpoint, and redirect attention to each other’s gestures.