Applying state-of-the-art artificial intelligence to grading in simulation-based education: assessment, feedback, and ROI

Accreditation bodies are driving competency-based education in healthcare, prompting curriculum reform. Simulation-based education (SBE) addresses challenges curriculum reform has uncovered, like lack of standardization in bedside teaching. This study explores the impact of an AI-powered Automated System Protocol (ASP) for grading students’ post-encounter notes in Clerkship OSCEs, comparing it to the legacy human grader system. The ASP, utilizing GPT-4, mapped rubric items to prompts. Analyzing post-encounter notes from 684 medical students across four academic years, we compared ASP with legacy Standardized Patient Evaluator (SPE) grades. Time efficiency, cost savings, and ROI analyses assessed educational and financial implications. Significant cost savings and efficiency gains were observed utilizing GPT-4 in comparison to SPEs. The Cost of Investment for ASP totaled $69,112 over 1,150 h. Comparing ASP to three SP graders yielded $13,112 in increased costs and initial time investment was required. However, beyond development time ASP execution-only, compared to legacy, showed an ROI of 589.44%, saving $47,877 with 87.5% time efficiency. ASP-execution versus three MD graders demonstrated an even stronger ROI of 797.09%. Implementing ASP in medical education provides substantial time and cost savings, enhancing ROI compared to legacy grading models. These findings highlight significant cost savings and efficiency improvements achievable through ASP implementation, positioning automated assessment as an innovative force shaping the future of medical education. By liberating human resources from manual grading and enhancing the immediacy of feedback, this approach contributes to a more efficient, effective, and engaging learning experience.