Lightweight multimodal EEG emotion recognition with cross-modal calibration for digital visual interaction

Objective: Digital visual interaction (DVI) systems, including virtual reality, augmented reality, and adaptive multimedia interfaces, demand real-time, lightweight emotion recognition to enable responsive and personalized user experiences. Multimodal approaches combining electroencephalography (EEG) and eye movement signals offer complementary neural and behavioral cues, yet existing Transformer-based methods restrict cross-modal interaction to final fusion layers, while heavier CNN-based alternatives incur prohibitive computational costs for interactive deployment. Methods: We propose the Cross-Modal Feature Calibration Multimodal Adaptive Emotion Transformer (CMFC-MAET), a lightweight framework tailored for DVI scenarios. CMFC-MAET extends MAET with two innovations: (1) Cross-Modal Feature Calibration (CMFC) modules inserted at intermediate Transformer layers, which generate channel-wise scaling and shifting factors from EEG features to calibrate eye movement representations; and (2) a Lightweight Gated Attention Fusion (LGAF) mechanism that incorporates feature statistics for adaptive multimodal integration. The model maintains an embedding dimension of 32 with only 3 Transformer layers and introduces only about 4.5K additional parameters over MAET, enabling real-time inference on consumer-grade hardware commonly used in DVI applications. Results: Extensive experiments on SEED, SEED-IV, and SEED-VII datasets demonstrate consistent improvements. On SEED-VII (7-class, subject-dependent), CMFC-MAET achieves 73.86% accuracy (a 2.58% gain over MAET). Cross-subject evaluation yields 43.12% accuracy (a 2.22% gain). Ablation studies confirm that CMFC contributes a 1.66% gain and LGAF contributes a 0.87% gain independently, while their combination produces the full 2.58% improvement with only about 4.5K additional parameters. Conclusion: CMFC-MAET achieves a strong accuracy-efficiency trade-off among compared multimodal emotion recognition methods, demonstrating that effective cross-modal calibration can be realized within a lightweight Transformer architecture suitable for real-time digital visual interaction systems.