Decoding an intuitive brain-computer interface combining speech and visual imagery using dual-branch network

As a new form of human-computer interaction, brain-computer interface (BCI) has been widely studied. However, most traditional BCI paradigms still face issues such as high cognitive load and easy fatigue. In recent years, intuitive BCI paradigms such as speech imagery (SI) and visual imagery (VI) have gradually become new research hotspots. Unfortunately, the classification performance of standalone SI and VI is still not ideal. In this paper, we designed an intuitive BCI paradigm combining SI and VI (SI+VI). Subjects are asked to simultaneously imagine images and Chinese characters associated with specific drone flight actions. Additionally, a dual-branch network using cross-attention mechanism to fuse time and frequency features (TFCA-DBNet) of electroencephalogram (EEG) signals is proposed. Experimental results show that the proposed TFCA-DBNet achieves a highest accuracy of 82.33% and an average accuracy of 67.33% in the four-class SI+VI task, which is improved by 15.80% and 6.90% compared to SI and VI, respectively. This indicates that the SI+VI paradigm has higher distinguishability than that of SI or VI alone. Compared with other methods, TFCA-DBNet also achieves better classification performance, which proves the effectiveness of the proposed method. Our study provides valuable reference and guidance for future applications of intuitive BCI paradigms.