Insights into the black box of multimodal meaning-making: investigating the reception of multimodality empirically

The intention of this article is to clarify the concept of multimodal meaning making from a recipient’s perspective on the basis of empirical data. The data comes from various reception studies in which different methods like eye tracking, knowledge tests, re-narrations, guided interviews or questionnaires were combined to investigate the appropriation of multimodal artifacts such as newspapers, science videos, comics, social media postings, or commercials. The article is structured in such a way that examples and results from various reception studies on newspaper pages, commercials, comics and science videos are presented in order to introduce concepts like attention, non-linearity, intermodality, modal density and modal coherence for clarifying the process of meaning-making. Investigating meaning-making empirically should not only fill a research gap in the mostly product- and production-oriented approaches to multimodality, but also help to develop a satisfactory theory of multimodality. Eye tracking in combination with verbal data reveals that multimodal artefacts have a non-linear structure – even in linear films – that is grasped step by step by the addressees, analogous to a hypertext. The concept of non-linearity opens a path to define the relevance structure of a multimodal arrangement: the relevance of modal elements is the outcome of its mutual contextualization. With this perspective on multimodality, the central unit of analysis is neither the sign-maker, the sign itself or its materiality but the interaction between communicator and addressees, for which modal resources are used. Intermodal relations are analysed not as constellations of signs, but as complex discursive actions addressed with a special intention to an audience or a person. The theoretical result can be summarized as follows: combining action theory with cooperation theory provides the conceptual tools to analyse meaning-making as a conversational implicature of a discursive action.