Examining Multisensory Integration in Semantically Incongruent Contexts: Evidence from Cross-Modal Mismatch Negativity and N400 Responses

Abstract Multisensory integration (MSI), in which information from different sensory modalities is processed in an interdependent manner, is often considered to occur automatically and pre-attentively. The present study aimed to investigate how modality weighting shapes the temporal dynamics of audiovisual integration during the pre-attentive stage using EEG. Twenty-two adults completed a novel audiovisual oddball task. Naturalistic animal images and sounds were presented as a frequent audiovisual standard, with visual, auditory, or audiovisual incongruencies presented as deviants. Mean amplitude analyses showed that visual and audiovisual deviants elicited significant frontocentral mismatch responses, whereas the auditory deviant did not produce a canonical mismatch negativity. A significant main effect of visual input on early ERP amplitude indicated that visual deviance exerted a stronger influence on early neural processing than auditory deviance. In contrast, auditory deviance was associated with delayed processing, particularly when paired with the expected visual stimulus, suggesting a temporal cost for non-dominant auditory information. At later stages, N400 responses emerged for auditory and audiovisual deviants but not for the visual deviant, suggesting greater downstream processing of non-dominant auditory information while visual mismatches may be resolved earlier. Together, these findings indicate that visual dominance influences the temporal organization of multisensory processing. Such temporal flexibility may be one way to facilitate the efficient interpretation of complex sensory input.