Research on multimodal interaction intention recognition based on EEG, eye movement, and gesture signals

Accurately recognizing interaction intention is crucial for enhancing the efficiency of human-computer interaction. Most current studies rely on single modalities or are confined to the fusion of eye movement and EEG signals, while the integration of gesture—a fundamental interaction modality—remains scarce, resulting in underutilized multimodal complementarity. Furthermore, existing methods support only a limited range of intention categories, restricting their applicability. To address these issues, this paper proposes a multimodal fusion recognition model that integrates EEG, eye movement, and gesture signals for recognizing six categories of interaction intentions. The model employs modality-specific networks to extract core features, a cross-modal attention mechanism to dynamically establish inter-modal dependencies, and a dynamic ensemble strategy to adaptively weight models based on performance. Evaluations on a public dataset show that the proposed approach outperforms traditional multimodal fusion models. Ablation studies confirm that both the attention mechanism and ensemble strategy are essential to the model’s performance. The research findings mitigate the shortcoming in the diversity of tasks for interaction intention recognition, expand the breadth of research in this domain, and provide both a theoretical foundation and technical support for the efficient recognition of multiple intention categories.