Decoding emotions through personalized multi-modal fNIRS-EEG Systems: Exploring deterministic fusion techniques

Emotion recognition through cortical neurovascular measurements poses significant challenges due to the complex interplay of emotions within cortical activity. Although Electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS) represent a promising combination for developing robust and accurate affective systems, the strategies for their integration remain relatively unexplored. This study aims to address this gap in personalized multimodal fNIRS-EEG emotion recognition systems. Confronted with the scarcity of large-scale multimodal fNIRS and EEG databases for emotion analysis, we developed our own dataset tailored for this research. We assessed widely used conventional models in single-modality emotion recognition to establish a benchmark. Various feature-level and decision-level fusion techniques were then implemented and evaluated against this benchmark. The multimodal fNIRS-EEG system consistently outperformed single-modality systems, achieving notable accuracy improvements of 7.5%, 3%, and 6.5% across the valence, arousal, and dominance dimensions, respectively. These improvements underscore the robustness of multimodal systems in emotion detection. Additionally, the top-performing feature-level fusion technique (direct concatenation) was on par with the best decision-level technique (average-based soft voting) across all dimensions. Our findings confirm the advantages of the multimodal fNIRS-EEG system over single-modality systems emphasizing on both individual characteristics of each modality and complementary nature of their hybrid system and suggest avenues for refining personalized emotion recognition approaches. This highlights the need for further investigation into optimized fusion techniques within alternative emotion recognition pipelines, paving the way for more nuanced and effective methodologies. • Integrating fNIRS and EEG enhances emotion accuracy across key dimensions. • Established a large-scale multimodal fNIRS-EEG dataset for affective analysis. • Analyzed feature and decision-level strategies in neuronal emotion recognition.