Visual cortex encodes spatially specific reward information during closed-loop naturalistic interaction

Our visual system actively gathers information from the environment to facilitate actions aligned with behavioral goals, and reward information plays a significant role in linking sensory inputs to optimal actions. While previous animal studies have demonstrated visual cortex’s sensitivity to rewards, the mechanism through which potential rewards modulate visual representations during goal-directed actions in dynamic naturalistic settings remains poorly understood, particularly in humans. In this study, we introduced an innovative naturalistic 3D interactive paradigm and examined the impact of rewards on visual representations. As part of the 7T Naturalistic Perception, Action, and Cognition (NatPAC) dataset, we collected human fMRI and high-precision eye-tracking data during a "shepherding" task in which participants continuously formulated plans and made actions within a Minecraft-based environment to herd as many sheep (i.e. potential rewards) as possible to a specified location while avoiding risks, such as puddles and a fox. We then used high-precision gaze data collected during the shepherding task, in conjunction with results from conventional population receptive field (pRF) mapping, to generate predicted BOLD signals based on gaze-centered visual inputs. Subtracting these predictions from actual BOLD signals removed the retinotopic visual stimulation effect, and the remaining residuals were modeled with reward-related regressors. To examine the spatial specificity of rewards, we segmented the visual field into eight radial bins and explained their variance with the reward-related regressors. The results showed significant spatial specificity for reward loss in V1, V2, and V3. In addition, correlation analysis revealed that higher reward-based spatial specificity in these regions correlated with more successful loss-avoidant behaviors. In summary, we discovered spatially specific reward signals in the visual cortex, which facilitate behaviors aimed at maximizing rewards. This study illustrates the intricate interplay of vision with action and cognition in naturalistic settings, which is rarely studied in conventional laboratory paradigms.