Mifu-ER: Modality Quality Index-Based Incremental Fusion for Emotion Recognition

Physiological information, such as that obtained from an electroencephalogram (EEG), electromyogram (EMG) or electrocardiogram (ECG), is often used as more objective and reliable data in emotion recognition studies because it is less affected by social masking. Recently, there has been a growing trend in emotion recognition studies to use multimodal data that fuses more than two physiological signals to improve emotion recognition performance based on this physiological information. This paper proposes a novel incremental fusion framework for multimodal physiological data-based emotion recognition, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</i>odality Quality Index (MQI)-based <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i>ncremental <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</i>usion for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">E</i>motion <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</i>ecognition (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Mifu-ER</i>). <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Mifu-ER</i> integrates three aspects: i) application of the sliding window method to ensure fast processing speed and a sufficient training dataset, ii) incremental fusion of physiological data by finding the MQI between modalities on the segmented dataset, and iii) emotion recognition by applying the fused multimodal dataset to the temporal convolution network (TCN) learning model. Experimental results show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Mifu-ER</i> achieves significant performance improvements in emotion classification, improving accuracies by 15% and 11% in subject-dependent and subject-independent learning, respectively, compared to single-modality approaches.