Prediction of negative user reactions towards system responses during attentive listening

Detecting user dissatisfaction with system responses is an important task for situated spoken dialogue systems. In this work we train models to predict negative user reactions which display this dissatisfaction. The dialogue scenario used in this work is attentive listening and we use two distinct corpora for analysis. We take a multimodal approach and use audio, visual and linguistic features in our model. Results show that audio features are the most influential and multimodality improves model performance. The importance of visual and linguistic modalities differ according to the type of corpus. We identify the key features of each corpus and find that a model trained on both corpora with seven key features reaches a precision of 0.562, outperforming the baseline for this task.