Assessing the Assessor: Machine Metacognition and Self-Assessment in an AI Coach

This paper explores an emerging dimension of AI-enhanced assessment practice - machine metacognition - through a case study of an adaptive coaching application built to operationalize the “design and systems thinking for transformation” (DOT) Framework. Traditionally, assessment has focused on evaluating human learning or performance, sometimes with the assistance of machines (e.g., automated scoring and, more recently, AI), while system performance of AI has been evaluated externally by human observers. In contrast, this study focused on an iterative process of human-prompted ‘machine self-assessments’ of the monitoring and control of the reasoning, responsiveness, and progression of an AI coach during cycles of test sessions with users. We analyze transcripts from a set of five cases to document the extent to which, and how, the Coach recognized, evaluated, and adjusted its actions across the stages of a variety of frameworks provided as contexts for generating coaching advice. Findings highlight evidence of what we have termed ‘machine metacognition,’ instances when the system, whether prompted or not, recognized misalignment, corrected course, or signaled uncertainty. We demonstrate how such contexts and mechanisms may redefine the boundaries of AI-enhanced assessment. The paper argues that recursive self-assessment within intelligent tools like the AI coach signals a conceptual shift toward double-loop assessment, in which both human and machine continuously learn about and from their evaluative processes.