Recommendations for Managing Ambiguities in Emotion Annotations

Emotion recognition is a growing area of research interest in multiple disciplines, including psychology, linguistics, and computer science. The field of affective computing has changed significantly with the advent of deep learning, which has brought in new and innovative techniques such as multimodal learning for emotion recognition. In this position paper, we explore the issues related to ambiguity in emotion annotations arising while designing and utilizing multimodal (specifically speech, vision and/or text) emotion-based datasets for applications modeling perceived human emotions. Our findings are based on observations and lessons learned while performing multimodal emotion recognition experiments focusing on oral history interviews. We categorize the ambiguity issues in emotion annotations into two categories, namely technical (emotion representation, contextual reference, segmentation, skewed annotator sample, and annotator evaluation) and socio-psychological (subject- and annotator-induced) factors. We provide eight recommendations for managing ambiguity in emotion annotation relevant for research on datasets for emotion recognition in the domain of affective computing.