How recording inputs affect multimodal AI agreement with reference scores in OSCE physical examinations: a retrospective ablation study
Grading physical-examination performance in objective structured clinical examinations (OSCEs) is labor-intensive, difficult to scale, and prone to inter-rater variability. In this retrospective ablation study of 300 OSCE encounters from six stations at the UT Southwestern Simulation Center, we examined how the agreement of a multimodal AI system (Gemini 2.5 Pro) with a composite reference standard changed as recording inputs were removed, and compared it with that of routine standard-of-care grading by non-physician standardized-patient evaluators (SPEs). Agreement was measured with quadratic-weighted kappa, and configurations were compared using paired learner-cluster bootstrap tests with false-discovery-rate correction. Against a composite reference standard with joint adjudication of discordant items by a physician and an emergency medical technician (EMT), three-camera native video achieved the highest agreement (quadratic-weighted κ = 0.832; 95% CI 0.775–0.885). Across isolated-modality configurations, agreement point estimates were ordered native video > audio-only > transcript-only > visual-only; the audio-only versus transcript-only difference was not statistically significant (raw p = 0.29; adjusted p = 0.36). Without audio, agreement dropped substantially and the system did not reliably distinguish performed from partially performed or omitted maneuvers (three-camera silent video κ = 0.239, 95% CI 0.167–0.307; full credit for 73.9% of items the reference scored below full credit). Agreement therefore depended largely on audio-inclusive inputs, which in OSCEs carry students’ verbal signposting and standardized-patient responses. Three-camera native video also agreed more closely with the reference than routine SPE grading (κ = 0.731; 95% CI 0.692–0.763; paired difference 0.10, 95% CI 0.04–0.16; unadjusted p = 0.003). For back range of motion, however, routine grading awarded full credit by instruction, whereas the reference applied the written rubric; without that item the difference was not statistically significant ( p = 0.17). These findings map recording requirements and limitations of AI-assisted physical-examination scoring in structured OSCEs. The locally constructed composite reference used non-physician SPE and EMT ratings; a physician and the EMT jointly adjudicated only discordant items, so the reference does not independently establish physical-examination competence. Generalization to unscripted clinical encounters remains to be tested.
Authors
- Michael J. Holcomb (ORCID: https://orcid.org/0000-0002-0595-1476)
- Huong-Tra Ngo (ORCID: https://orcid.org/0000-0001-5016-3391)
- Philip C. Jarrett (ORCID: https://orcid.org/0000-0003-4534-8556)
- Ameer Hamza Shakur (ORCID: https://orcid.org/0000-0002-2695-8378)
- Thomas O. Dalton (ORCID: https://orcid.org/0000-0002-8351-8336)
- Andrew Raymond Jamieson (ORCID: https://orcid.org/0000-0001-5416-9379)
- David Hein (ORCID: https://orcid.org/0000-0002-8625-9528)
- Hunter Schuler (ORCID: https://orcid.org/0009-0002-6162-9198)
- Shinyoung Kang
Institutions
- The University of Texas Southwestern Medical Center (US)
Publication Details
- Journal
- BMC Medical Education
- Published
- 2026-11-13
- DOI
- https://doi.org/10.1186/s12909-026-10779-6
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00