Assessing the integration of social information using AI-generated videos: a brief research report

Integrating social information from different cues and modalities is a central feature of social functioning and humans respond sensitively to incongruences between different social information types (e.g., smiling while uttering a negative statement about oneself). To investigate the processing of these integration processes, researchers frequently draw on the simultaneous or consecutive presentation of static stimuli, such as written text and images, which often exhibit a relatively low naturalness of these stimuli. In this brief research report, I illustrate how AI-generated videos can successfully be used to overcome some of these limitations. Videos were generated by combining different avatar images (that were also generated with an AI tool) with text messages that either presented statements that were congruent or incongruent with expectations based on the age, gender, or facial emotion expressions of the avatar. In this pilot study, 47 participants watched these videos and rated the plausibility and emotional intensity of each utterance. Despite being fully generated artificially, videos elicited different responses depending on the condition. Incongruences based on social categories (age, gender) engendered lower plausibility ratings while intensity ratings were relatively unaffected. In contrast, intensity ratings for emotion expressions were expectedly higher (irrespective of congruence) but sensitivity to plausibility was affected by the congruence of the facial expression and the statement of the avatar. This pilot study illustrates the potential of AI-generated stimuli for investigating socio-cognitive processes, particularly in research on the integration of social information.