Semantic and Visual Features Drive the Intrinsic Memorability of Co-Speech Gestures

Co-speech gestures that teachers spontaneously produce during explanations have been shown to benefit students’ learning. Further, prior work suggests that information conveyed through teachers’ gestures is less likely to deteriorate than through speech (Church et al., 2007). However, how intrinsic features of gestures affect students’ memory remains unclear. The memorability effect denotes a phenomenon where adults with different backgrounds consistently remember and forget particular visual stimuli (static images, dance moves, etc.), owing to the stimuli's intrinsic semantic and visual features. In this study, we investigate whether certain gestures are consistently remembered and, if so, which semantic and visual features are associated with these remembered gestures. We first created 360 10-second audiovisual stimuli by video recording 20 actors producing unscripted natural speech and gestures as they explained Piagetian conservation problems. Two trained experimenters extracted high-level semantics and low-level visual/acoustic features in speech and gesture for each audiovisual stimulus. We then tested online participants’ memories in three different conditions using a between-subjects study-test paradigm: the audiovisual stimuli (gesture+speech condition), the visual-only version of the same stimuli (gesture condition), and the audio-only version of the stimuli (speech condition). Within each of the two experimental blocks, participants encoded nine random stimuli from an actor and made old/new judgments on all 18 stimuli from the same actor immediately after. We discovered that participants show significant consistencies in their memory for the gesture, gesture+speech, and speech stimuli. Focusing on the visual-only (gesture) condition, we found that (1) more meaningful gestures and speech predicted more memorable gestures; (2) using both hands led to more memorable gestures than using one hand. Our results suggest that both semantic (conveyed through speech and gestures) and visual (conveyed through gesture) features make co-speech gestures memorable.