Verbal and facial expression cues in emotional alignment

Emotional alignment during conversation, defined as the mutual recognition of and exchange of appropriate emotional expressions between interlocutors, is essential for successful interactions, and can be elicited by both verbal content and facial expressions. But do facial expressions produced during conversation correspond more directly to the emotion expressed through the interlocutor’s words or their facial movements? Are different emotions, expressed via facial expressions and storytelling, equally influential on listeners’ emotional processing? To address this, we examined emotional alignment during dyadic video-mediated conversations, using listeners’ facial mimicry and pupil dilation as measurable indices of voluntary and autonomic alignment, employing an experimental design that included three emotional polarities and two conditions for manipulating the match between verbal and displayed facial expressions. Our results showed that facial expressions have a greater impact on autonomic reactions than verbal emotional expressions. Also, similar to previous studies on in-person conversation, smiles induce pupil dilation, whereas negative facial expressions induce pupil contraction. Our study also found that lip-area movements associated with negative facial expressions are mirrored earlier and more often. This new discovery on the role of negative emotion in early emotional alignment could have important implications for better understanding the nature of the phenomenon.