Classifying interpersonal synchrony in VR joint actions: human-human versus human-bot interactions

Understanding how interpersonal synchrony emerges in immersive virtual environments is essential for modeling joint action and extending social interaction to artificial agents. However, objective computational methods capable of classifying spontaneous synchrony from minimal sensor data remain limited. This study collected sensor data from VR goggles in pairs and examined interpersonal synchrony using a within-subject design. Participants positioned opposite each other in VR space judged object categories (kitchen vs. garage utensils) and responded with movements. We created an LSTM classification model using only non-verbal head-motion acceleration data from pairs instructed to interact either competitively (Game session) or collaboratively (Collab session), without incorporating any linguistic or verbal communication cues. The model achieved 93.2% accuracy (F1 score = 0.932) in classifying these interaction modes using only three-axis head acceleration data from the goggles. We then applied this within-subject trained model to Bot interaction sessions, where participants worked with Bot avatars under high or low accuracy conditions. Dynamic Time Warping (DTW) distance was used to quantify synchrony. Results showed that when movements were classified as synchronous, DTW distance was lower, indicating higher synchrony. A significant interaction effect was found between movement classification (Game/Collab) and Bot accuracy condition on DTW distance. Exploratory cross-subject validation showed reduced performance (F1 = 0.72), though head rotation features remained important across participants. These findings demonstrate that the classification model successfully captures interaction dynamics across different task conditions within subjects, and that synchrony with Bots is modulated by behavioral predictability.