Multimodal Emotion Prediction in Conversations via Conditional Variational Generating Network and Uncertainty learning

Multimodal emotion prediction in conversations keeps being a challenging subject in artificial intelligence which aims to forecast the emotion type of the following dialogue before it takes place. Most researchers achieve impressive results by applying emotion analysis model directly. However, emotion prediction needs to classify the future emotion with corresponding information completely unknown, which is actually largely different from the traditional emotion analysis problem. To compensate for this deficiency, a novel model based on a conditional variational generating neural network is specially designed for multimodal emotion prediction. An uncertainty learning layer is also added to improve the predictive performance. Concretely, three utterances containing three modalities are assembled as a new multimodal input unit. A conditional variational auto-encoder based on trasformer is followed to generate prospecting following target unit on both latent and feature level by only using history unit features. The final classifier will obtain predictive results through Gated Recurrent Units(GRU) decoder based on generated information and historical contexts, in which an uncertainty layer is integrated. Experiments are carried out on two widely used datasets which show better results than recent works.