Development and feasibility of an AI-driven immediate feedback system for observation-based clinical placements: A design-based research study

INTRODUCTION: Reflection is essential for professional development in clinical education; however, students in early-stage observation-based clinical placements may adopt passive learning approaches, while limited supervisory capacity can restrict timely, individualized formative feedback. This study aimed to develop an artificial intelligence (AI)-driven immediate feedback system for daily reflective notes, examine its feasibility, and explore short-term patterns in rubric-based reflective writing. METHODS: Using a design-based research framework, we implemented the system with 89 undergraduate students in a judo therapy program during a four-day observation-based clinical placement. Students submitted daily reflective notes, completed rubric-based self-assessments, and received AI-generated rubric scores and feedback within one minute. Feasibility was assessed using system performance, submission and completion rates, and student perceptions. As an exploratory external check, 120 reflective notes from 30 randomly selected students were independently rated by three blinded clinical educators using the same rubric. RESULTS: All 356 system requests were processed successfully. Daily reflective note submission rates exceeded 98%, and self-assessment completion rates exceeded 95% on all days. AI-generated total scores increased from Day 1 to Day 2 and remained at similar levels thereafter. Blinded human ratings showed a moderate rank association (Spearman's ρ = .495) but limited absolute agreement (intraclass correlation coefficient = .399) with AI-generated total scores. The early increase in AI-generated total scores was not reproduced in human total ratings, although human ratings of observational specificity increased from Day 1 to Day 4. Student questionnaire responses indicated favorable perceptions of the system's usability and feedback. DISCUSSION: The system showed stable operation and high student engagement, supporting its feasibility for providing timely, rubric-based formative feedback under clinical educator oversight. However, AI-generated scores should be interpreted as system-internal indicators rather than independent evidence of educational improvement. Further controlled studies using external outcome measures are required.