Semantic Reconstruction from Fnirs Using Recurrent Neural Networks

Semantic reconstruction of language aims to decode the meaning of words or sentences from neural activity. Previous studies have demonstrated that functional near-infrared spectroscopy (fNIRS) contains information suitable for language decoding. However, most of the existing work on fNIRS-based neural decoding relies on traditional machine learning algorithms such as linear models and support vector machines, and it has been limited to classification of limited set of words. To address these shortcomings, we examine 4 recurrent neural networks (RNNs) that learn features to decode semantic representations from fNIRS: the Elman recurrent neural network (ERNN), long short-term memory (LSTM), and bidirectional version of them (BiERNN and BiLSTM). Using a publicly available fNIRS dataset, we performed within-category, between-category, and leave-two-out tests. The decoding performance was measured by computing the matching score, a pairwise metric that assesses the model's ability to distinguish between two concepts. The results show that ERNN and BiLSTM models consistently outperform linear decoder models. Specifically, ERNN shows better performance for 4 out of 7 subjects in the between-category test, and BiLSTM performs better for 6 out of 7 subjects in the within-category test and 4 out of 7 subjects in the leave-two-out test. Notably, in between-category experiment, the BiLSTM scored 61 % matching score for subject 3, representing a 9% improvement, and ERNN achieved an 80% matching score for subject 2, marking a significant 33% improvement. These promising results encourage the use of advanced machine learning models for semantic reconstruction from fNIRS. Code is available at https://github.com/sposso/Semantic-Reconstruction-using-fNIRS-signal.