Crossmodal counterpoint: from music to multimedia – incongruency, cognitive dissonance, irony, and surrealism

Laboratory-based research on multisensory perception often presents participants with unpredictable combinations of auditory and visual stimuli that may be classed (by the experimenter) as either congruent or incongruent. Cognitive neuroscientists generally assume that congruent combinations of experimental stimuli will be processed more fluently and lead to enhanced crossmodal binding and multisensory integration than will incongruent combinations of stimuli. Typically, however, the participants involved in such laboratory research are given little if any information (or context) to explain why these seemingly random combinations of sensory stimuli are being presented. This situation differs markedly from the deliberate combination of eye and ear in an artistic context (say when music is added to film). In the latter case, conflict is sometimes introduced deliberately into a scene. A film director might, for example, choose to combine violent onscreen action with uplifting happy music. The presentation of such audiovisual emotional incongruency invites the viewer to reflect on what they are experiencing, and why this particular combination of stimuli has been chosen. Such crossmodal counterpoint (or cognitive dissonance) is sometimes used as a rhetorical device to introduce a note of irony. It is interesting to note how, in such cases, there is little sense of averaging the sensory inputs (or their emotional effects) as is so often seen when congruent inputs are presented in cognitive psychology studies. In this narrative historical review, we take a critical look at the concept of crossmodal counterpoint, and review the research and theorizing on its use.