Onderzoek van de Overeenkomst tussen Convolutionele Neurale Netwerken en de Hersenen bij Sociale Cognitie

Artificial Neural Networks (ANNs) attempt to reproduce 'in silico' processes that take place in the human brain. In the domain of AI, ANNs are now the state-of-the-art method for tackling a variety of problems (computer vision, speech recognition, natural language processing, etc.). Here, the similarity to the human brain is mostly incidental: the brain may serve as a source of inspiration, but it is really the performance of the ANN that matters, e.g., how well it can classify images or understand speech. On the other hand, ANNs are also a popular tool in neuroscientific studies. Here, in stark contrast to the AI domain, similarity to the human brain is key. The ANN is of interest only insofar as it can help to better understand the human brain. This PhD is situated within the interdisciplinary project 'Computational Modeling of Social Cognition and associated Deficits by means of Artificial Neural Networks' (KU Leuven IDN/21/010) that brings together people from Computer Science, Psychology and Neuropsychiatry to address one of the biggest challenges in both AI and neuroscience: understanding social cognition. The concept of social cognition comprises multifaceted interpersonal processes that underly the interpretation of other people's behaviors and how to interact with them. In particular, this thesis asks the question: do Convolutional Neural Networks (CNNs), a common type of ANN that has achieved remarkable successes in many computer vision tasks, form a good basis to model the human brain for the specific task of social cognition? Indeed, the successes of CNNs have mainly been booked on low-level tasks, such as, e.g., identifying objects in pictures. Social cognition goes beyond such low-level tasks by necessitating the integration of multiple information streams, and potentially calling upon prior, latent knowledge. Nevertheless, earlier work has established certain correspondences between the visual processing stream in the human brain and CNNs, making them an ideal starting candidate to model this task. The motivation for this work resides in the fact that how social cognition manifests in the brain is not entirely understood yet. In particular, it is not well understood how neurodivergent brain structures—either through lifelong conditions such Autism Spectrum Disorder or neurodegenerative conditions such as Frontotemporal Dementia—affect social skills. If a computational model were to be found that manages to capture the essence of neurotypical social cognitive processing, this model could then potentially be used to investigate how altered brain mechanisms, reflected in altered network structures, affect social cognition. This type of research, whereby computer models are used to model the brain, is generally referred to as Computational Psychiatry or Computational Neuroscience. The present dissertation opens with a short, non-technical introduction to this field. Following this, a novel dataset for 'Emotion Recognition in the Wild'—i.e., using non-stages scenes—called FindingEmo is described, including an analysis of why previously existing datasets were insufficient for the task at hand. Using this dataset as a starting point, a method, EmoCAM, is introduced that allows to investigate the importance of specific object classes for the detection of specific emotions by a CNN. Essentially, the question is asked: What information does the network use to make its assessment? Next, making use of both FindingEmo and EmoCAM, the question of to what extent CNNs align with the human brain for the task of valence appraisal is explored by performing a correlation analysis with fMRI data obtained from neurotypical individuals. Some attention is also devoted to a limitation of CNNs, and how despite showing similarities with human visual processing they also show weaknesses not exhibited by humans, in the subsequent chapter focused on how color alterations alter CNN predictions. Finally, the FWO ID-N project 'Social cognition in silico and in vivo' is briefly described. This project forms a direct continuation of, and will build directly on results presented in this dissertation.