Emotion Recognition Using a Siamese Model and a Late Fusion-Based Multimodal Method in the WESAD Dataset with Hardware Accelerators

Emotion recognition, which includes stress analysis, is a critical research topic. In particular, emotion recognition using wearable applications that offer rapid feedback can significantly assist in emotion regulation. A key dataset for this purpose is the wearable stress and affect detection (WESAD) dataset, which is well-suited for wearable environments. This study aims to evaluate the accuracy and processing time of the proposed neural network using the WESAD dataset. However, data collected in wearable environments often have limitations in terms of their accuracy and channel count, making classification challenging. To address this issue, this study utilizes only wrist-measured blood volume pulse (BVP) data measured by photo-plethysmography (PPG) and electro-dermal activity (EDA) data, classifying them through a late fusion-based multimodal method. For accurate classification, the BVP data are preprocessed using the empirical mode decomposition (EMD) method, and a Siamese model is employed for learning on small datasets, with an additional structure of dense layers for detailed classification. The Siamese network generates a similarity score, which is then fed into a dense layer to classify multiple classes. Meta-classes are used to compute pairwise similarities, ultimately classifying the data into one of four emotional states: neutral, stress, amusement, and meditation. The proposed multitasking model simultaneously trains the Siamese network and multi-classifier, achieving an accuracy of 99.8% in classifying four emotion states in the WESAD dataset. Moreover, the model is designed to operate with minimal hardware resources, making it easy to implement using hardware accelerators. This study demonstrates the applicability of neural network-based wearable systems in the field of emotion recognition.