Prosodic synchrony and the semantic affiliation of gestures in the multimodal utterance

In the last decade, much research in the Autosegmental framework (Loehr 2012; 2014; Rohrer et al. 2023; Rohrer 2022) has highlighted a strong synchronization between Strokes' apex and prosodic prominences (pitch accents and prosodic edges). The paper addresses whether the synchrony with prosodic prominence signals the gesture’s semantic affiliation. The question is studied through the fine-grained analysis of an interview given by an actor where gestures were previously annotated (Cantalini 2022). The detection of prominences is achieved following the Language into Act Theory (Cresti 2000; Moneglia & Raso 2004), which connects Perceptively Relevant Prosodic Contours (‘t hart et al. 1990) to the pragmatic functions of the Information Units (IUs). Even assuming this framework, the synchronization of gestures to prosodic prominence is systematic. The paper shows that the affiliation of gestures must concern the IU where the prominence falls (neither backward nor forward). Prosodic synchrony, however, does not determine the type of semantic affiliation, which can be lexical, modal, or pragmatic; it does, however, impose substantial restrictions. The stance of modal gestures can scope on one word or one IU. Pragmatic gestures correlate with the pragmatic functions foreseen by L-AcT and synchronize with prominences bearing the corresponding value. IUs not bearing a functional prominence (Scanning IU) can affiliate lexical or modal but never pragmatic gestures.