Multi-modal fusion network with intra- and inter-modality attention for sleep stage classification

Sleep is one of the most basic human physiological activities, and sleep assessment based on physiological time series can effectively help human experts diagnose people's sleep health. Although existing methods have achieved good results, it remains a challenge to fully utilize the heterogeneity and correlation of multiple modalities to improve sleep staging performance. Although existing methods have achieved good results, different modalities such as electroencephalogram (EEG), electrooculogram (EOG), and electromyogram (EMG) differ significantly in their data characteristics, scales, and types of information, and it remains a challenge to fully utilize the heterogeneity and correlation of multiple modalities to improve sleep staging performance. Therefore, this paper proposes a multimodal fusion method that considers cross-modality and intra-modality features simultaneously. First, a dual-stream feature extractor extracts EEG, EOG, and EMG sleep features. Then, the inter-modality fusion module is designed to capture the correlation and complementarity of the features between two modalities using cross-attention. Meanwhile, the intra-modality encoding module was added to keep the heterogeneity information of each feature. Finally, the obtained correlation and complementarity features between the two modalities are fused with the heterogeneity features of the respective modalities and delivered to the classifier for classification. In addition, this paper uses temporal continuity encoding to encode the temporal relationships of time series signals in the Transformer. We evaluate our model on two public datasets: ISRUC-S3, and Sleep-EDF-20. Our model achieves 82.83% and 87.83% accuracy on the above two datasets. The experimental results show that our method is comparable to existing methods on two datasets of different sizes.