Multimodal Brain Signal Analysis with State-Space Modeling: A Study on Working Memory

Working memory (WM) is a fundamental cognitive function intricately linked to various neurological and psychiatric conditions. In this study, we introduce Neural-SSM, a novel computational model based on the State Space Model (SSM), designed to effectively capture and characterize neural activity patterns. The model was trained on multimodal brain signals, specifically electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) data collected during the N-back tasks. Experimental results indicate that the Embedding Unit achieved an accuracy of 81.41% in classifying N-back task difficulty, while the Attention GRU component demonstrated precise regression of hemodynamic responses with a mean squared error (MSE) of 0.1289. Analysis of the attention matrix revealed enhanced directional connectivity from EEG to fNIRS channels and identified long-range pathways from the occipital to frontal cortex, which intensified with increasing task difficulty. These findings highlight the model’s potential in advancing the understanding of brain activities in cognitive processes, with significant implications for biomedical engineering and neuroimaging research.