Self-Reported Sentiment Estimation with Attention Mechanism to Integrate Time-Series Physiological Signals andWord Sequences

One of the main issues in the development of an adaptive dialogue system is to estimate a user’s sentiment state since the user’s self-reported sentiment does not necessarily appear in the user utterances. To mitigate the issue that the true sentiment state is not expressed as observable signals, psychophysiology and affective computing studies have focused on physiological signals that capture involuntary changes related to emotions. We address the issue by proposing a new attention mechanism based on the time-series physiological signals and word sequences. Our proposed method, called Time-series Physiological Transformer which captures sentiment changes based on both linguistic and physiological information, significantly outperformed the previous best result (p < 0.05).