Multiheaded Attention based Convolution Neural Network Towards Emotion Classification using Multimodal Dataset

Emotion is an indispensible element in the daily human conversations and it helps to understand behaviors. Humans express different emotions in the conversations. Thus recognizing the emotion in the conversation is challenging task. In recent decade, Emotion Recognition has gathered wide attention among many researchers towards analyzing and detecting emotion in the conversations using unimodal information such as text, speech, facial expressions and gestures. In order to achieve those outcomes, artificial intelligence architectures based on machine learning architectures and deep learning architectures has been employed to extract the feature and aggregating them together for recognition task is performed. Despite of several advantages, those architecture leads to several challenges and it has been described in detail. In this paper, a new Multihead attention based spatial and temporal attention has been incorporated to convolutional neural network as it determines the weighted features effectively through incorporation of the attention coefficient to process the text, video, audio information in the form of tensor. In convolution layer, relevant feature is extracted using kernel function and it is organized in form of feature map. Attention mechanism is incorporated in the pooling layer to select the emotion specific feature on refining the feature map to generate the attention map. Those feature attention and its map is processed in the fully connected layer using batch normalization function, activation function before classifying the feature in the softmax function. Softmax function uses the naive bayes classifier to classify the disease feature into heart disease classes based on condition of heart. Finally loss function using cross entropy and epoch value were set to eliminate the over fitting and class imbalance challenges. Experimental and performance analysis of the multiheaded attention based convolution neural network against traditional deep learning model for emotion recognition has been performed using CMU-Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset. Data is partitioned into training and testing data for architecture training and model evaluation using test data. Finally test data is processed through confusion matrix in form of cross fold validation to estimate the precision, recall, accuracy. It is proved that currentmultiheaded attention based convolution neural network towards multimodal emotion recognition outperforms other multimodal emotion recognition architectures.