BrainLM: Enhancing Brain Encoding and Decoding Capabilities with Applications in Multilingual Learning

With the rapid advancement of large-language models in natural language processing (NLP), many studies have explored their role in brain encoding and decoding. In this study, we developed BrainLM, a pre-trained multimodal model that incorporates paired brain activity data from text stimuli. We demonstrated its accuracy in brain encoding and decoding across multiple NLP tasks. Our research produced several notable findings: we successfully developed a model for brain encoding and decoding, validated its reliability through bidirectional experiments, and outperformed 20 state-of-the-art models in brain encoding tasks. Additionally, we designed an autoencoder module to extract brain features. We extended the capabilities of BrainLM to new datasets and explored multilingual tasks using transfer learning, which enhanced the generalization ability of the model. Notably, BrainLM achieved 51.75% accuracy in binary classification tasks and increased the correlation coefficient by 3%–15% in brain prediction tasks. This study expands the applications of BrainLM and uncovers the complex interactions between brain regions and language models across different linguistic environments.