A riot of blooms begins to dazzle the eye: cognitive behavior with multimodal discourse during usefulness judgments of health information

Purpose As the two sides of the same coin, usefulness and usability have emerged as pivotal research themes in user experience field. This study compares cognitive effort and cognitive resource allocation strategy across documents varying perceived usefulness and then across documents with different objective usability (unimodal vs multimodal discourses). Design/methodology/approach A controlled user study of four identifying tasks related to public health epidemics was conducted to collect data, including document usefulness as perceived by participants, presentation modes of the document and gaze behaviors on each document. Findings Usefulness and modality discourse impact cognitive effort and resource allocation strategy in health information search. In useless health documents, spatial encoding resource spending increased significantly with multimodal discourse, and a spatial browsing strategy with an evident exploratory feature was applied; while in useful documents, including low-useful and high-useful, both spatial and information encoding resource spending increased significantly with multimodal discourse, and an information processing strategy with an evident comprehensive feature was applied. Notably, multimodal discourse failed to enhance decision-making effectiveness. Furthermore, in useful documents, the interaction effect of the presentation mode of useful information and multimodal discourse on cognitive effort followed an inverted U-shape pattern. Originality/value This paper sheds new light on the interaction effect of usefulness and usability on cognitive effort and resource allocation strategy, highlighting its significance in cognitive effort detecting for multimodal discourse and improving effectiveness and efficacy of health information identification by optimizing information presentation mode design.