MTSNet: Convolution-Based Transformer Network With Multi-Scale Temporal-Spectral Feature Fusion for SSVEP Signal Decoding

Improving the decoding performance of steady-state visual evoked (SSVEP) signals is crucial for the practical application of SSVEP-based brain-computer interface (BCI) systems. Although numerous methods have achieved impressive results in decoding SSVEP signals, most of them focus only on the temporal or spectral domain information or concatenate them directly, which may ignore the complementary relationship between different features. To address this issue, we propose a dual-branch convolution-based Transformer network with multi-scale temporal-spectral feature fusion, termed MTSNet, to improve the decoding performance of SSVEP signals. Specifically, the temporal branch extracts temporal features from the SSVEP signals using the multi-level convolution- based Transformer (Convformer) that can adapt to the dynamic fluctuations of SSVEP signals. In parallel, the spectral branch takes the complex spectrum converted from temporal signals by the zero-padding fast Fourier transform as input and uses the Convformer to extract spectral features. These extracted temporal and spectral features are then integrated by the multi-scale feature fusion module to obtain comprehensive features with different scale information, thereby enhancing the interactions between the features and improving the effectiveness and robustness. Extensive experimental results on two widely used public SSVEP datasets, Benchmark and BETA, show that the proposed MTSNet significantly outperforms the state-of-the-art calibration-free methods in terms of accuracy and ITR. The superior performance demonstrates the effectiveness of our method in decoding SSVEP signals, which may facilitate the practical application of SSVEP-based BCI systems.