Encoding Affective Cues in Multimodal Textual Transcriptions

Multimodal annotations provide important cues for understanding how a conversation proceeded, particularly in relation to affective factors. In this paper, we extend the automated conversation annotation system MONAH with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pitch</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">volume</i> annotations to encode new affective cues, making MONAHv3 the state-of-the-art automatic annotation system in terms of the number of automatically annotated aspects. MONAHv3 provides an automated solution that, while not a direct equivalent, offers performance that is competitive with the manually produced Jefferson transcription system. In automatic evaluations, the additions significantly improve supervised learning in ten out of eighteen experiments. In human evaluations of emotion recognition, the additions significantly outperformed the Jefferson transcription system. Usability studies further showed that MONAHv3 is much more user-friendly than Jefferson-style transcripts. Lastly, in human evaluations of paralinguistic cues (e.g., tone and volume), MONAHv3 achieved performance comparable to the Jefferson system, while remaining more competitive in kinesics (e.g., describing actions). This distinction between automated and manual systems is critical for appreciating the scalability and accessibility that MONAHv3 offers.