Gaze reinstatement during working memory for natural scenes

Abstract The information we hold in mind with working memory (WM) may propagate beyond the cortex and out to the peripheral motor system. For instance, directionally biased hand and eye movements can express simple visuo-spatial features during WM maintenance. If such sensorimotor inflections are a form of WM representation, they may also express more multifaceted and task-adaptive information than just spatial location. Here, we ask whether gaze patterns carry item-specific detail about complex stimuli held in mind, and whether gaze prospectively adapts to the behavioral task demands. To do this, we tracked human eye movements (male and female) during WM for naturalistic images, and we manipulated which image features were most task-relevant (visual or semantic). In two experiments, we found that the eyes moved in complex, item-specific spatiotemporal sequences during WM maintenance. Delay period gaze patterns retraced the scanpaths from stimulus encoding, echoing the ‘gaze reinstatement’ that aligns with neural pattern reactivation during long-term memory retrieval. Therefore, gaze patterns during WM may reinstate visual encoding processes for maintenance. We also used time-resolved representational similarity analyses, guided by convolutional neural network modeling, to test the specificity of WM gaze patterns across a trial. We found that gaze representations were more distinct when the task prioritized visual detail, and also ramped up in anticipation of the WM probe. Therefore, oculomotor WM signals are malleable to when and how the content will be used. These results suggest that the earliest levels of visual processing reflect prospectively-oriented and functionally flexible WM content representations. Significance The eyes are an outwardly accessible extension of the brain. They are the first stop for visual processing, but may also offer a rich window into underlying cognition. For instance, pupil dilation can reflect changing cognitive load, and eye movement biases can reflect basic spatial information about content held in mind with working memory. Working memory is a critical neuro-cognitive function, but its physiological bases remain under debate. Here, we show that eye movements code for more than simple spatial WM, and instead express intricate patterns that read out the complex, real-world stimuli held in mind. This work shows how cognitive states influence oculomotor functions, and also advances theories of WM representation to include the earliest stages of sensory processing.