ChatGPT as a Competent Enough Judge in Evaluating Relevance of Responses from Divergent Thinking Task

The validation of responses in divergent thinking tasks is a critical yet understandardized step that should precede creativity scoring. However, inconsistencies related to human judges in this step may compromise the reliability of the results. This study introduces a systematic approach using ChatGPT to validate responses in the Alternate Uses Task (AUT) and compares its performance against six human judges. Analyzing 1245 AUT responses for common objects, we evaluated validity based on precisely defined criteria. Human judges exhibited significant variability, achieving unanimous agreement for only 58% of responses, while ChatGPT demonstrated significant alignment with human assessments, reflecting a capacity to replicate aggregated human judgment. These findings underscore the potential of Large Language Models to enhance objectivity and reproducibility in creativity research by automating response validation. We advocate for integrating AI-driven validation protocols into divergent thinking response evaluation and emphasize transparent reporting of criteria to advance methodological rigor in the field. Full text: https://escholarship.org/uc/item/0fs5t29x This work was supported by the National Science Center, Poland [Grant Number: 2021/41/N/HS6/01620]