Investigating the Effects of Sleep Conditions on Emotion Responses with EEG Signals and Eye Movements

Existing studies in psychology and neuroscience have extensively examined the effects of sleep deprivation on emotional responses. More recently, researchers have begun applying deep learning algorithms to further investigate this relationship, emphasizing the importance of accessible and high-quality multimodal datasets across different sleep states. To address this need, we develop SEED-SD, a multimodal dataset comprising data from 40 participants. The dataset includes electroencephalography (EEG) and eye movement signals collected under three sleep conditions: sleep deprivation (SD), sleep recovery (SR), and normal sleep (NS). Each condition contains data corresponding to four basic emotions: happiness, sadness, fear, and neutral state. Additionally, we propose a novel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Re</b>gion Transformer with <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</b>ayer-<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b>usion (ReLF), to conduct comprehensive analyses on the SEED-SD dataset. ReLF incorporates a region- wise self-attention mechanism to extract localized features from EEG and eye movement signals, and supports flexible adaptation to both multimodal and unimodal inputs. Following multimodal generative pre-training, ReLF introduces learnable prompts to replace the missing modality under unimodal settings, thereby enabling effective fine-tuning of the pre-trained model. The experimental results demonstrate that ReLF outperforms the existing models. Our analysis further reveals that SD significantly impacts emotion recognition performance, while SR and NS conditions yield similar results, highlighting the importance of SR in mitigating the adverse effects of SD. Furthermore, we conduct a systematic analysis of multimodal complementarity, critical frequency bands, and neural patterns. Our findings reveal distinct EEG patterns under the SD condition compared to the SR and NS conditions. Notably, the multimodal complementarity and critical frequency bands in both the SD and SR conditions align with those observed in the NS condition. In summary, to the best of our knowledge, SEED-SD is the largest publicly available multimodal dataset for studying the relationship between emotion recognition and sleep states. This dataset lays a crucial foundation for applying deep learning methods in this area. Moreover, through extensive data-driven analysis, this work confirms the inhibitory effect of SD on emotion recognition and the restorative role of SR. The SEED-SD dataset and codes will be public upon paper acceptance.