The Impact of a Hidden AI-Based Chatbot on the Quality of Collaborative Problem Solving in a School Context

The increasing use of digital devices by young learners often results in passive content consumption rather than active skill development. This exploratory study examines whether a peer-like Artificial Intelligence (AI) agent can improve the quality of computer-supported collaborative learning. The aim was to assess the impact of a hidden AI-based chatbot on the dynamics and outcomes of group problem-solving in a school setting. A gamified application was developed in which student groups collaborated on challenging tasks. In a controlled experiment, some groups included a hidden AI-based chatbot acting as a peer, programmed to provide Socratic prompts and motivational scaffolding without giving direct answers, while control groups consisted only of human participants. Quantitative and qualitative data, including time to solution, answer correctness, and chat logs, were collected to compare performance and interaction patterns between the two conditions. Given the limited sample size and primarily descriptive analyses, the findings should be interpreted as preliminary. The results suggest differences in collaborative dynamics and problem-solving efficiency between groups assisted by the AI agent and the unassisted control groups. The findings suggest that integrating a hidden, peer-like pedagogical agent may represent a promising approach for supporting collaborative learning processes, enhancing group engagement by subtly guiding discussion without disrupting the natural peer-to-peer dynamic. These results highlight the potential of hidden AI to enhance collaborative learning environments through non-intrusive support. Further research with larger samples is needed to validate these initial observations.