A Symbolic State-Space Framework for Quantifying the Dynamical Structure of Multimodal Conversational Interaction

Abstract Human conversation depends on the sequential coordination of multiple communicative channels. We present a general methodological framework for quantifying the dynamical organisation of multimodal conversational behaviour. Conversational behaviour is represented as trajectories through a discrete, symbolic state space that combines speech, gaze, backchannels, and laughter from both interlocutors. We apply a set of dynamical and information-theoretic analyses, namely recurrence quantification analysis (RQA), cross-recurrence analysis, information-theoretic measures (block entropies, entropy rate, excess entropy), Markov transition summaries, and a leave-one-covariate-out (LOCO) procedure for quantifying modality-specific contributions to conversational structure. As a proof-of-concept application, using a corpus of dyadic conversations recorded across three conversational contexts (introduction, a collaborative tangram task, and open discussion), we show that conversational trajectories exhibit clear recurrence structure alongside substantial local variability. Structure was systematically modulated by task demands: the most constrained collaborative task produced the strongest recurrence and the lowest entropy. Modality-specific analyses further showed that speech and gaze were the dominant structure-contributing channels, whereas backchannels and laughter contributed only weakly to global dynamical organisation. We describe the encoding, metrics, and statistical procedures so that the framework can be applied to other multimodal interaction datasets.