EOG and PPG Fusion for Subject Independent Multimodal Emotion Recognition: A Prototypical Networks Approach

Multimodal emotion recognition is recognizing the emotion from more than one modality like facial expression, gestures, speech, text and physiological signals. The prototypical networks is a well-known few-shot distance based machine learning algorithm, in which data point is classified based on the proximity to the prototype. In this work, the prototypical network algorithm is modified for multimodal emotion recognition using EOG and PPG, in which Smooth L1 loss is explored for learning the embedding space. This study also explored correlation, Euclidean, squared Euclidean and chord distance as distance metrics to learn the embedding space. The proposed method obtained a prediction accuracy of 60.00% and 61.00% in arousal and valence dimensions respectively which is higher than the accuracy reported in the literature.