Understanding the Recognition of Dynamic Multimodal Expressions of Single and Blended Emotions

Nonverbal emotion expressions are multimodal patterns of behavior that unfold dynamically. However, several key questions remain about the nature of dynamic multimodal emotion expressions. This thesis aimed to better understand how such nonverbal expressions—both single and blended emotions—are recognized. Study I investigated individual differences in the ability to recognize single emotions in the general population using the ERAM test. Study 1 focused on emotion recognition in relation to emotional competencies, personality, and socio-emotional dysfunction, and evaluation of the psychometric properties of the ERAM test. Study 2 examined emotion recognition in relation to metacognitive judgments using an online version of the ERAM test. Overall, results suggested that emotion recognition is related to empathy, emotion understanding, openness, and alexithymia, and also with metacognitive skills. The results further revealed that accuracy was highest in the multimodal condition and positively correlated across modalities. Lastly, no differences were found between the in-lab and online versions of the test, indicating that the ERAM can be reliably administered online. Study II investigated the ability to recognize blended emotions (pairwise combinations of 5 emotions). Recordings of actors displaying two emotions in equal proportions (50:50 expressions) were used. The test combined a rating scale and a forced-choice task, and participants were instructed to choose 2 out of 5 available scales. Study 1 examined multimodal emotion recognition, whereas Study 2 examined emotion recognition in unimodal conditions (video-only and audio-only). Recognition accuracy was highest when expressions were presented multimodally, and also higher in the video-only condition, consistent with the findings on single emotions in Study I. Both studies further showed that all combinations were recognized above chance level, regardless of presentation modality. Study III investigated the recognition of blended emotions with varying proportions (70:30, 50:50, 30:70 expressions), using recordings from the 6 best-recognized actors in Study II. Study 1 examined emotion recognition using a restricted response format (prompted to use 2 out of 5 scales, as in Study II), while Study 2 used an unrestricted version of the same test (free to choose any number of the 5 scales). The results showed that the majority of blends across all proportions were recognized above chance level in both formats (restricted vs. unrestricted). Results further revealed that the more prominent emotion received higher ratings than the less prominent in most combinations. These findings replicated and extended those in Study II, and suggest that both the quality and quantity of emotions can be recognized. Together, the results showed that, in the general population, individual differences in emotion recognition of single emotions are related to broader affective, personality, and metacognitive processes. They also revealed that recognition accuracy of both single and blended emotions is highest when dynamic multimodal expressions are presented multimodally. This thesis contributes to a growing body of work that underscores the significance of studying emotions conveyed dynamically through the face, voice, and body, and emphasizes the need to increase the number and complexity of emotions under study, as emotion recognition ability appears to be more nuanced and flexible than previously thought.