Uncertainty Weighting as a Time-Invariant Principle of Crossmodal Inference

In a dynamic world, events unfold and disappear, often providing only fleeting or incomplete sensory evidence. Information from another sense may help us interpret these encounters, but what determines how strongly it influences what we later remember? Across two audiovisual working-memory experiments, we tested whether auditory context exerts greater influence when visual information is reconstructed less reliably. Participants viewed a face morphed along a feminine–masculine continuum for 100ms and, after a 1,900-ms retention interval, reproduced it using a continuous face-morph response wheel. On cue trials, a feminine- or masculine-sounding voice was presented 1,000ms before face onset (prediction) or 900ms after face offset (postdiction). In Experiment 1 (N=84), voice cues shifted reconstructions toward the cued gender in both tasks. Across observers, weaker visual sensitivity was associated with greater auditory influence; and within observers, faces reconstructed less consistently without cues showed stronger bias. Experiment 2 (N=170) extended this relationship to externally imposed visual uncertainty through resolution reduction and spatial occlusion. Across both manipulations and cue timings, less reliable reconstruction was associated with greater auditory cue influence. Despite this consistency in uncertainty-sensitive auditory influence, the way voices influenced reconstructions depended on whether they were heard before or after the face. Together, these findings suggest a recurring functional principle of crossmodal inference: The less reliably visual information can be reconstructed, the more strongly contextual information influences the resulting judgment. Differences in how auditory bias was expressed suggest prediction and postdiction may address the same inferential challenge through different underlying processes.