Decoding Hostility from Conversations Through Speech and Text Integration

Hostility is a complex trait with emotional, cognitive, and behavioral components. Hostility identification in conversational or transactional scenarios can benefit healthcare systems by, for example, predicting cardiovascular disease risks. While conventional hostility assessment relies on interviews, training proficient interviewers and mitigating biases pose significant challenges. In response, this study introduces the GMGF-MIL method to pioneer automatic multi-modal hostility detection in a structured interview. This approach utilizes recurrent neural networks to capture conversational context while integrating a graph neural network-based technique to merge acoustic and textual data. Furthermore, attention-based multiple-instance learning pooling is employed to aggregate utterance-level information. Notably, this is the first paper to introduce a novel multi-modal automated conversational hostility assessment approach, filling a notable gap in existing resources. Our evaluations showcase the efficacy of the GMGF-MIL method, achieving an accuracy of 78% in distinguishing between high- and low-hostile individuals.