Language Models and Brain Alignment: Brain Encoding and Decoding

Can we obtain insights about the brain using AI models? How is the information in deep learning models related to brain recordings? Can we improve AI models with the help of brain recordings? Such questions can be tackled by studying brain recordings like functional magnetic resonance imaging (fMRI). As a first step, the neuroscience community has contributed several large cognitive neuroscience datasets related to passive reading/listening/viewing of concept words, and narrative stories. Encoding and decoding models using recent advances in deep learning have opened new opportunities for modelling brain activity and exploring the convergent representations underlying language comprehension in the human brain and in neural language models (LM). Using encoding and decoding, what insights can we draw from recent, largely task-free neuroimaging datasets for theories of language and the brain? This tutorial will provide a working knowledge of stimulus representations, popular naturalistic neuroscience datasets, and state-of-the-art methods for brain encoding and decoding. It will also offer a better understanding of the benefits and limitations of encoding and decoding with deep learning.