CNN-Transformer Based Temporal-Spatial Fusion Network for EEG-fNIRS Hybrid Decoding

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are two widely used modalities in brain-computer interface (BCI) systems. However, decoding performance based on a single modality remains limited. Integrating both modalities and leveraging their complementary strengths offers a promising approach to enhance neural signal decoding. In this paper, we propose a dual-branch multimodal neural network for EEG-fNIRS hybrid decoding. Convolutional neural networks (CNNs) and Transformer architectures are employed to extract rich temporal and spatial features from each modality. EEG and fNIRS signals are processed through separate branches, and their extracted temporal-spatial features are subsequently fused for hybrid decoding. Extensive experiments on three public datasets demonstrate the superiority of the proposed method and its potential for practical BCI applications.