Modulating Multisensory Processing: Interactions Between Semantic Congruence and Temporal Synchrony

Presenting information to multiple sensory modalities often facilitates or interferes with processing, yet the mechanisms remain unclear. Using a Stroop-like task, the two reported experiments examined how semantic congruency and incongruency in one sensory modality affect processing and responding in a different modality. Participants were presented with pictures and sounds simultaneously (Experiment 1) or asynchronously (Experiment 2) and had to respond whether the visual or auditory stimulus was an animal or vehicle, while ignoring the other modality. Semantic congruency and incongruency in the unattended modality both affected responses in the attended modality, with visual stimuli having larger effects on auditory processing than the reverse (Experiment 1). Effects of visual input on auditory processing decreased under longer SOAs, while effects of auditory input on visual processing increased over SOAs and were correlated with relative processing speed (Experiment 2). These results suggest that congruence and modality both impact multisensory processing.