Improving Speech Emotion Recognition with Emotion Dynamics

Existing speech emotion recognition (SER) research is developed assuming static human emotions. However, emotions are dynamic in nature. In this work, we use IEMOCAP databases and modify the conventional SER systems by capturing the intra-utterance temporal dynamics of human emotions. We explore signal-level and feature-level descriptors of emotion variability and compute the utterance-wise mean-squared-successive-difference (MSSD). The MSSDs are used to recognize the utterances as low or high inertia. Emotion inertia defines the trait of emotion propagating from one time to the next. The emotional states in the low inertia utterances exhibit frequent transitions. We show that such transitions prominently degrade the conventional SER system’s performance. In our work, we detect the low inertia utterances during evaluation and proposed methodologies to nullify the stray emotion transitions. The modifications improve the overall performance by 2.87%.