How recording inputs affect multimodal AI agreement with reference scores in OSCE physical examinations: a retrospective ablation study

Grading physical-examination performance in objective structured clinical examinations (OSCEs) is labor-intensive, difficult to scale, and prone to inter-rater variability. In this retrospective ablation study of 300 OSCE encounters from six stations at the UT Southwestern Simulation Center, we examined how the agreement of a multimodal AI system (Gemini 2.5 Pro) with a composite reference standard changed as recording inputs were removed, and compared it with that of routine standard-of-care grading by non-physician standardized-patient evaluators (SPEs). Agreement was measured with quadratic-weighted kappa, and configurations were compared using paired learner-cluster bootstrap tests with false-discovery-rate correction. Against a composite reference standard with joint adjudication of discordant items by a physician and an emergency medical technician (EMT), three-camera native video achieved the highest agreement (quadratic-weighted κ = 0.832; 95% CI 0.775–0.885). Across isolated-modality configurations, agreement point estimates were ordered native video > audio-only > transcript-only > visual-only; the audio-only versus transcript-only difference was not statistically significant (raw p = 0.29; adjusted p = 0.36). Without audio, agreement dropped substantially and the system did not reliably distinguish performed from partially performed or omitted maneuvers (three-camera silent video κ = 0.239, 95% CI 0.167–0.307; full credit for 73.9% of items the reference scored below full credit). Agreement therefore depended largely on audio-inclusive inputs, which in OSCEs carry students’ verbal signposting and standardized-patient responses. Three-camera native video also agreed more closely with the reference than routine SPE grading (κ = 0.731; 95% CI 0.692–0.763; paired difference 0.10, 95% CI 0.04–0.16; unadjusted p = 0.003). For back range of motion, however, routine grading awarded full credit by instruction, whereas the reference applied the written rubric; without that item the difference was not statistically significant ( p = 0.17). These findings map recording requirements and limitations of AI-assisted physical-examination scoring in structured OSCEs. The locally constructed composite reference used non-physician SPE and EMT ratings; a physician and the EMT jointly adjudicated only discordant items, so the reference does not independently establish physical-examination competence. Generalization to unscripted clinical encounters remains to be tested.

Authors

Institutions

Publication Details

Journal
BMC Medical Education
Published
2026-11-13
DOI
https://doi.org/10.1186/s12909-026-10779-6
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

How recording inputs affect multimodal AI agreement with reference scores in OSCE physical examinations: a retrospective ablation study

Michael J. Holcomb, Huong-Tra Ngo, Philip C. Jarrett, Ameer Hamza Shakur et al.
3 links
BMC Medical Education
Artificial Intelligence in Healthcare and Education
article

How recording inputs affect multimodal AI agreement with reference scores in OSCE physical examinations: a retrospective ablation study

Michael J. Holcomb, Huong-Tra Ngo, Philip C. Jarrett, Ameer Hamza Shakur, Thomas O. Dalton, Andrew Raymond Jamieson, David Hein, Hunter Schuler, Shinyoung Kang
article en

Abstract

Grading physical-examination performance in objective structured clinical examinations (OSCEs) is labor-intensive, difficult to scale, and prone to inter-rater variability. In this retrospective ablation study of 300 OSCE encounters from six stations at the UT Southwestern Simulation Center, we examined how the agreement of a multimodal AI system (Gemini 2.5 Pro) with a composite reference standard changed as recording inputs were removed, and compared it with that of routine standard-of-care grading by non-physician standardized-patient evaluators (SPEs). Agreement was measured with quadratic-weighted kappa, and configurations were compared using paired learner-cluster bootstrap tests with false-discovery-rate correction. Against a composite reference standard with joint adjudication of discordant items by a physician and an emergency medical technician (EMT), three-camera native video achieved the highest agreement (quadratic-weighted κ = 0.832; 95% CI 0.775–0.885). Across isolated-modality configurations, agreement point estimates were ordered native video > audio-only > transcript-only > visual-only; the audio-only versus transcript-only difference was not statistically significant (raw p = 0.29; adjusted p = 0.36). Without audio, agreement dropped substantially and the system did not reliably distinguish performed from partially performed or omitted maneuvers (three-camera silent video κ = 0.239, 95% CI 0.167–0.307; full credit for 73.9% of items the reference scored below full credit). Agreement therefore depended largely on audio-inclusive inputs, which in OSCEs carry students’ verbal signposting and standardized-patient responses. Three-camera native video also agreed more closely with the reference than routine SPE grading (κ = 0.731; 95% CI 0.692–0.763; paired difference 0.10, 95% CI 0.04–0.16; unadjusted p = 0.003). For back range of motion, however, routine grading awarded full credit by instruction, whereas the reference applied the written rubric; without that item the difference was not statistically significant ( p = 0.17). These findings map recording requirements and limitations of AI-assisted physical-examination scoring in structured OSCEs. The locally constructed composite reference used non-physician SPE and EMT ratings; a physician and the EMT jointly adjudicated only discordant items, so the reference does not independently establish physical-examination competence. Generalization to unscripted clinical encounters remains to be tested.

BMC Medical Education
The University of Texas Southwestern Medical Center (US)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon