Deciphering visual representations behind subjective perception using reconstruction methods

Reconstruction techniques have been widely used to recover physical sensory inputs from brain signals. Numerous studies have consistently refined methods to achieve image reconstruction that faithfully mirrors the presented image at the pixel level. An intriguing extension of these techniques is their potential application to subjective mental contents, a domain that has proven to be especially challenging. Here, we introduce a general framework that can be used to reconstruct subjective perceptual content. This framework translates or decodes brain activity into deep neural network (DNN) representations, and then converts them into images using a generator. Through our research on visual illusions—a classic form of subjective perception defined by a discrepancy between sensory inputs and actual perception— we demonstrate how we successfully reconstructed visual features that were absent in the sensory inputs. Our work shows the potential of reconstruction techniques as invaluable tools for delving into visual mechanisms. The use of natural images as training data and the choice of DNNs were key in obtaining successful reconstruction. While extensive research has probed the neural underpinnings of visual illusions using qualitative hypotheses, our approach materializes mental content into formats amenable to visual interpretation and quantitative analysis. Reconstructions from individual brain areas shed light on the strength of illusory representation and its shared representations with real features at different levels of processing stages, which provides a means to decipher the visual representations underlying illusory perceptions.