Hierarchical Spatiotemporal Attention Network for Fine-grained Brain Cognitive State Recognition

Brain cognitive state recognition based on functional Magnetic Resonance Imaging(fMRI) can capture brain functional activities under different tasks and help understand the neural mechanisms of the brain, which has always been one of the focuses of neuroscience research. Different from the prediction of the brain cognitive domain, the prediction of brain fine-grained cognitive state is based on each moment in the process of executing the task. Therefore, it is necessary to extract more fine-grained effective information. Existing studies focus on modeling and classifying the complete time series, ignoring the brain activity state at each time. So we propose a hierarchical spatiotemporal attention network(FineBrainNet) to recognize fine-grained brain cognitive state. Guided by coarse-grained cognitive domain labels, we trained different sub-modules for fine-grained states under each cognitive domain to capture relevant cognitive state changes more accurately in specific task. Extensive experiments on the HCP-Task dataset show that FineBrainNet can achieve accurate prediction of fine-grained brain cognitive state.