Talk It Out: Concurrent Versus Retrospective Think-Aloud for Complex Verbal Assessment Tasks

Think-aloud methods provide a window into test-takers' real-time cognitive processes, offering valuable evidence for validity based on response processes. Yet, their use in complex verbal assessments remains limited, particularly concerning the choice between concurrent and retrospective protocols. This multiple-method study directly compared these two approaches in the context of a high-stakes verbal selection test, involving 10 biomedical students equally divided between conditions. Using a combination of deductive coding (based on the assessment blueprint) and inductive thematic analysis, we explored the richness and nature of participants' explanations. Quantitative analysis (interpreted cautiously due to the small sample) further supported the conclusions from the qualitative findings. Participants in the concurrent condition verbalized more frequently, often reading items aloud and articulating their reasoning with greater clarity and depth. In contrast, retrospective participants tended to offer shorter, more fragmented responses, with less transparency in their thought processes. Quantitative results revealed higher assessment scores and significantly more verbal engagement in the concurrent group. Moreover, participants reported the concurrent think-aloud as easier and more natural to perform. These findings may challenge the prevailing assumption that concurrent think-aloud is unsuitable for verbal tasks due to cognitive load. Instead, our results suggest that it can elicit richer, more authentic process data, even in complex verbal assessment contexts, although the preliminary nature of the current study must be emphasized. This study offers a practical contribution to the collection of response process evidence, with clear implications for both researchers and practitioners aiming to enhance the validity and design of educational assessments.