Unveiling passive cross-modal reactivation and validation processes in the processing of multimedia material

During reading, a passive reactivation process reactivates information from the discourse context, while a passive validation checks the consistency between reactivated and actual information. Two pre-registered online studies examined whether passive cross-modal reactivation and validation processes occur during the processing of multimedia materials (text combined with pictures). Participants were recruited via Clickworker (Experiment 1: N = 251; Experiment 2: N = 266). We applied the epistemic Stroop paradigm (Richter et al., 2009), which utilizes automatic validation mechanisms that trigger positive or negative response tendencies. These tendencies can interfere with an unrelated task, requiring an opposite response. Participants were shown matching (valid) or mismatching (invalid) text-picture stimuli, with the text and picture components presented sequentially, interrupted by a calculation task, making reactivation necessary. Following each stimulus, participants performed an unrelated probe-word task responding to the words "right" or "wrong". Dependent variables were reaction time and error rates in the probe-word task. Experiment 1 used one-sentence-picture stimuli, while Experiment 2 used longer text segments with pictures. In both experiments, linear mixed-effects models revealed interactions between validity (valid vs. invalid stimulus) and probe word (right vs. wrong) for reaction times and error rates. Post-hoc analyses showed longer reaction times or higher error rates when participants encountered invalid stimuli before responding to the probe word “right”. In this study, passive cross-modal reactivation and validation processes could be demonstrated together for the first time in multimedia processing. Implications for understanding information processing in multimedia contexts are discussed. • During processing multimedia, passive cross-modal reactivation processes occur. • During processing multimedia, passive cross-modal validation processes occur. • The reactivation mechanism reactivates information from the discourse context. • The validation mechanism checks reactivated and current information consistency. • These mechanisms occur with one-sentence-picture and longer-text-picture stimuli.