Semantic-based audiovisual integration enhances active storage in visual working memory

Presenting a semantically congruent sound during visual target encoding can facilitate subsequent short-term memory retrieval, which is generally considered as a beneficial effect of multisensory integration on working memory. However, there remains a fundamental doubt about whether this benefit in performance is truly due to semantic-based multisensory integration rendering information about the target object stored more precisely in working memory, or simply due to better activation of the object's archetypal template preexisting in long-term memory. The present event-related potential (ERP) study investigated this issue in humans with a fine-grained visual object recognition task to prevent any stronger activation of the object's internal template from affecting performance. Meanwhile, a retro-cue was introduced to enable the dissociation of the ERPs reflecting audiovisual interactions and the contralateral delay activity (CDA) indexing active visual working memory maintenance. Our Experiment 1 unveiled that the congruent-sound-induced benefit to recognition performance was accompanied by an increased CDA, both of which could be predicted by an early ERP correlate of semantic-specific audiovisual interaction. Experiment 2 further revealed that these behavioral and electrophysiological effects were present only when the congruent sound was delivered synchronously with, but not prior to, the memory items, invalidating cross-modal semantic priming as an alternative account. These findings offer the first direct evidence that multisensory semantic integration enables details about the target object itself to be stored more precisely in working memory, hence the improved short-term memory performance, which establishes a theoretical foundation for improving working memory with multisensory integration.