Beyond scenario creation: Human and GenAI synergy in situational judgment test response development

Self-regulated learning (SRL) is required for learners of all ages and disciplines. There is a need for a variety of assessment tools to assess the professional knowledge of teachers in SRL and self-regulated teaching (SRT). Situational judgment tests (SJT) can serve as a measurable assessment tool for evaluating teachers' SRL and SRT knowledge. This research constitutes the second stage in the development of such an SJT tool dealing with the formulation of responses to scenarios created in the previous stage. Additionally, the introduction of GenAI-assisted evaluation poses the need to explore the still understudied human-GenAI interaction during assessment tools development. This study used thematic analysis for exploring the responses generated by GenAI (ChatGPT) and researchers for the previously formulated scenarios to examine the contribution of both agents in the tool development process. A main theme – standardization and patterning (GenAI) versus flexibility and openness (researchers) was found. The study highlights the potential of combining GenAI and researchers in developing assessment tools and the contribution of each.