Generating objects in peripheral vision using attention-guided diffusion models

Despite the majority of our visual field being blurry in the periphery, with only the central ~2 degrees offering high-resolution inputs, we have no difficulty perceiving and interacting with objects around us. We hypothesize that the human perception of a stable visual world is mediated by an active generation of objects from blurred peripheral vision. Furthermore, we hypothesize that this active peripheral generation is task-dependent, guided by information extracted from fixations, with the goal of constructing a relevant object and scene context for the current task. We test these hypotheses by using latent diffusion models and evaluating the influence of fixated image information on generating objects in the blurred periphery. We ask this question in the context of an object referral task, in which participants hear a spoken description of the search target (e.g., “right white van”). We recorded eye movements from participants (n=220) as they viewed 1,619 images and attempted to localize the referred targets. The model received high-resolution input only from fixated regions, mimicking foveated vision, and generated high-resolution objects in the originally blurred peripheral areas. We found that using foveated-image inputs corresponding to observed behavioral fixations led to the model generating target objects in the periphery with greater fidelity compared to randomly located fixations, as measured by squared pixel difference (Human Fixation SSE = 178.27; Random Fixation SSE = 212.42; averaged over the first 20 fixations). This fixation-driven advantage specifically applied to the reconstruction of task-relevant objects, such as objects of the same referred category, and did not extend to non-targets or background elements. Our findings support the idea that human perception actively generates relevant objects in the blurry periphery as a means of building a stable object context, which is guided by goal-directed attention control mechanisms.