SSTMNet: Spectral-Spatio-Temporal and Multiscale Deep Network for EEG-Based Motor Imagery Classification

Motor impairment is a critical health issue that restricts disabled people from living their lives normally and with comfort. Detecting motor imagery (MI) in electroencephalography (EEG) signals can make their lives easier. There has been a lot of work on detecting two or four different MI movements, which include bilateral, contralateral, and unilateral upper limb movements. However, there is little research on the challenging problem of detecting more than four motor imagery tasks and unilateral lower limb movements. As a solution to this problem, a spectral-spatio-temporal multiscale network (SSTMNet) has been introduced to detect six imagery tasks. It first performs a spectral analysis of an EEG trial and attends to the salient brain waves (rhythms) using an attention mechanism. Then, the temporal dependency across the entire EEG trial is worked out using a temporal dependency block, resulting in spectral-spatio-temporal features, which are passed to a multiscale block to learn multiscale spectral-–spatio-temporal features. Finally, these features are deeply analyzed by a sequential block to extract high-level features, which are used to detect an MI task. In addition, to deal with the small dataset problem for each MI task, the researchers introduce a data augmentation technique based on Fourier transform, which generates new EEG trials from EEG signals belonging to the same class in the frequency domain, with the idea that the coefficients of the same frequencies must be fused, ensuring label-preserving trials. SSTMNet is thoroughly evaluated on a public-domain benchmark dataset; it achieves an accuracy of 77.52% and an F1-score of 56.19%. t-SNE plots, confusion matrices, and ROC curves are presented, which show the effectiveness of SSTMNet. Furthermore, when it is trained on augmented data generated by the proposed data augmentation method, it results in a better performance, which validates the effectiveness of the proposed technique. The results indicate that its performance is comparable with the state-of-the-art methods. An analysis of the features learned by the model reveals that the block architectural design aids the model in distinguishing between multi-imagery tasks.