A Theory-Driven Game-Based Assessment of Collaborative Problem-Solving

Collaborative problem-solving (CPS) is an essential 21st-century competency, yet its valid and ecologically sound assessment remains challenging due to inconsistent frameworks, poorly aligned task design, and over-reliance on behavior frequency counting in scoring. This study developed and provided empirical support for a theory-driven game-based CPS assessment embedded in a real-time strategy game, using a customized Collaborative Siege Task with asymmetric roles and ill-structured problems, to elicit more realistic collaborative interactions. Based on a validated three-facet CPS framework from Chinese cultural context, the task systematically triggers social and cognitive processes of collaboration across five progressive stages. By sampling from the eastern provinces of China, a pilot study with 36 participants confirmed task feasibility and indicator effectiveness, while a formal study with 298 valid participants examined psychometric properties using item response theory and confirmatory factor analysis. Findings supported the hierarchical higher-order factor structure, satisfactory item difficulty, discrimination, reliability, and criterion-related validity against peer assessment, reasoning ability, and personality. The assessment showed no sex, major, or role bias. This study provides a reliable gamified assessment tool for CPS measurement with demonstrated potential for scalability pending further development of automated scoring procedures, offering implications for large-scale evaluation and competency-oriented instruction.