The Effect of Aural and Visual Presentation Formats on Categorical and Dimensional Judgements of Emotion for Sung and Spoken Expressive Performances

The purpose of this study was to investigate the effects of visual information on the perception of emotion in three contexts: a short spoken phrase, an analogous short melody, and a longer melody with greater complexity of pitch and rhythm. Participants without substantial formal music training were assigned to either an audio-only, visual-only, or audiovisual presentation mode; all observed musicians and actors who sang melodies or spoke phrases intending to communicate happiness, sadness, or anger. Participants rated these performances for positive and negative valence, energy arousal, and tension arousal. They were also asked to select the discrete emotion they perceived, and to rate how certain they were about this selection. Participants perceived energy arousal and tension arousal as fairly distinct features of the performances and the performances were perceived as having clear dimensional characteristics based on intended emotion (e.g., happy performances had the highest positive valence, lowest negative valence, etc.). Regarding presentation, participants in the audiovisual mode categorized emotional intentions with greater accuracy than those in the others. Although ratings of negative valence, energy, and tension ratings were more extreme for actors than for musicians, participants were most in agreement about the valence of the actors’ sung performances.