Enhancing cybersecurity via an emotion aware framework: leveraging EEG and speech signal fusion through correlation-based networks

This research explores the enhancement of cybersecurity systems by integrating emotion-based techniques with conventional brain-mapping measurements. By incorporating biosignals such as electroencephalography (EEG) and speech analysis, this approach allows for a more comprehensive evaluation of emotional and cognitive states, improving the reliability and robustness of lie detection. Despite the lack of external proof from such evaluations, the proposed system represents a significant advancement over conventional methods, offering deeper insights into user intentions and emotion levels. This paper presents a method for multimodal emotion detection using physiological and behavioral signals—such as EEG and speech expressions; however, several challenges arise in integrating these modalities effectively. These include the complexity of synchronizing multiple data sources, applying appropriate fusion techniques, and using advanced representation learning to extract meaningful emotional cues. Overcoming these issues is crucial to fully realizing the potential of this enhanced cybersecurity framework. The proposed model was evaluated using the newly introduced EAV and PME4 datasets, both well suited for multimodal emotion analysis owing to their synchronized EEG and speech expressions. The modality-specific weighted feature-fusing strategy was utilized to combine the features extracted from the DCCA network, and the fused features were then used to train the suggested ANN classifier. The proposed model achieved performances of 68.95% and 78.67% for the four-emotion categories in the PME4 and EAV datasets respectively.