Analyzing Communication Logs in Pair Programming: A Comparison of Human- and LLM-Based Approaches

Communication challenges have often been a significant b arrier t o e ffective P air Programming (PP), especially for novices in higher education. A deep understanding of communication patterns can enhance learning outcomes during PP. To explore this, we conducted an experiment involving 19 participants engaged in debugging tasks at a university, grouped into three pairing configurations: e xpert p airs, student pairs, and mixed pairs. We manually transcribed and coded the participants' verbal interactions based on nine predefined communication patterns. Considering that manual coding is cost-intensive, we also explored an automated annotation approach by leveraging recent Large Language Models (LLMs) with zero-shot capabilities for multi-label classification. Our findings revealed distinct differences in communication patterns. Integration, extension, feedback request, and critique were the most common patterns, while completion, justification request, clarification,j uxtaposition, a nd p araphrase w ere r are a cross all groups. These insights highlight the importance of fostering a comfortable and supportive environment that encourages agreement and idea expansion during PP, particularly those that require collaborative programming practices. Furthermore, our model evaluation indicates that the advanced GPT-4o model performs best, achieving a F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf>-score of 0.59. This study suggests that encouraging diverse transactive interactions can enhance the effectiveness of PP. Additionally, the LLM-based automated annotation approach shows promise as a substitute for human observers, prompting large-scale communication research.