Predicting Teamwork Performance Using a Large Language Model to Annotate Repair and Grounding Utterances in Task-Based Conversations

Effective teamwork relies heavily on the quality of communication among team members. Conversational dynamics, such as the use of repair and grounding mechanisms—conversational strategies that maintain mutual understanding—play a crucial role in promoting cohesion within a team. Here, we describe an approach that leverages a large language model (LLM) to detect repair and grounding (R&G) utterances during a cooperative task and evaluate how these factors can predict team performance. To demonstrate this, we collected and analyzed video data from YouTube of the cooperative multiplayer puzzle game Keep Talking and Nobody Explodes. Player communication was transcribed using speech-to-text tools to generate speaker-labeled transcripts. Utterances were labeled for R&G mechanisms by an LLM using few-shot learning. Statistical analyses reveal distinct patterns in communication between successful and unsuccessful bomb defusal trials, which enabled the development of a model to predict task outcome.