A Multimodal Deep Learning Framework for Emotion-robust Biometric Identification

The role of emotion-robust biometric identification is increasingly important because identity cues should remain reliable under changes in the affective state of a user. This article introduces a multi-modal framework that re-interprets the AFFEC dataset, which was initially created to analyse emotions, as a benchmark for biometric identification of 73 subjects across five synchronized modalities, namely, EEG, face, eye tracking, galvanic skin response and cursor behaviour. To prevent leakage and evaluate cross-session generalization, a stringent run-wise protocol is adopted that converts raw recordings into identification-oriented descriptors and raw EEG tensors based on modality-specific preprocessing. Classical baselines such as support vector machine, random forest, multilayer perceptron and k-nearest neighbors are compared to three deep-learning architectures: an EEGNet-based dual-stream model, Resnet based dual-stream model which uses a hybrid with a residual network, and a CNN-Transformer hybrid. Experimental results demonstrate that EEG and face are most discriminative modalities, and their combination makes the best performance, with EEGNet achieving 98.38% test accuracy of EEG+Face and 97.84% test accuracy of full multimodal fusion. The research shows that multimodal fusion is better at enhancing robustness whereas emotion-robust learning may also help close the divide between affective variability and biometric stability.