A brief peer-assessed OSCE for evaluating undergraduate abdominal ultrasound training: a three-cohort psychometric study

Abstract Background Much of undergraduate ultrasound teaching takes place in student-led, near-peer courses, but learners’ skills are rarely re-assessed once such a course is established. Full objective structured clinical examinations (OSCEs) often exceed the resources of these programmes, and brief peer-assessed formats need evidence before their scores can be interpreted. We examined which interpretations are justified, and for which purpose. Methods Psychometric observational study of three consecutive cohorts (2025 to 2026) of a three-day, student-led basic abdominal ultrasound course for fifth-semester medical students at one German faculty. Participation was voluntary. Each student completed two 10-minute stations on two of seven randomly allocated organs, scored on 13 criteria (maximum 14 points) by trained student tutors; a subsample was rated independently by two examiners. The primary question was whether aggregated criterion- and organ-level results are stable enough to inform course review. Supporting objectives concerned the interpretation of individual scores, the relation to a 15-item knowledge test and delivery. Results One hundred fifty-three students were scored on at least one station and 136 on both (total score 20.7 ± 4.1 of 28). The single-rater intraclass correlation coefficient (ICC) was 0.78 (95% CI 0.63–0.88; n = 39) and 0.73 (0.52–0.86; n = 33), and 0.65 and 0.69 after adjustment for organ; within-station Cronbach’s alpha was 0.70 and 0.69, and 0.34 for the four interpretive criteria. Dynamic scanning criteria were hardest and among the most discriminating, basic criteria showed ceiling effects, and the difficulty ranking was concordant across cohorts (Kendall’s W = 0.92). The allocated organ accounted for 22% of score variance, examiners for 7% in a crossed model. Between-station consistency was low ( r = 0.17; organ-adjusted ICC 0.35); a reliability of 0.70 would require 5 organ-standardised stations (range 2 to 10). The total score correlated with the post-course knowledge test ( r = 0.38; n = 79). Conclusions Aggregated criterion- and organ-level results were reproducible across cohorts and can inform iterative review of a student-led course, within limits of precision, representativeness and transferability. Individual total scores from two stations are not reliable enough for consequential decisions about learners, regardless of agreement between examiners. Readiness for independent clinical scanning is not addressed.

Authors

Institutions

Publication Details

Journal
BMC Medical Education
Published
2026-11-16
DOI
https://doi.org/10.1186/s12909-026-10782-x
Primary Topic
Innovations in Medical Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A brief peer-assessed OSCE for evaluating undergraduate abdominal ultrasound training: a three-cohort psychometric study

Florian Recker, Severin Pfeil, Mario Iasevoli
2 links
BMC Medical Education
Innovations in Medical Education
article

A brief peer-assessed OSCE for evaluating undergraduate abdominal ultrasound training: a three-cohort psychometric study

Florian Recker, Severin Pfeil, Mario Iasevoli
article en

Abstract

Abstract Background Much of undergraduate ultrasound teaching takes place in student-led, near-peer courses, but learners’ skills are rarely re-assessed once such a course is established. Full objective structured clinical examinations (OSCEs) often exceed the resources of these programmes, and brief peer-assessed formats need evidence before their scores can be interpreted. We examined which interpretations are justified, and for which purpose. Methods Psychometric observational study of three consecutive cohorts (2025 to 2026) of a three-day, student-led basic abdominal ultrasound course for fifth-semester medical students at one German faculty. Participation was voluntary. Each student completed two 10-minute stations on two of seven randomly allocated organs, scored on 13 criteria (maximum 14 points) by trained student tutors; a subsample was rated independently by two examiners. The primary question was whether aggregated criterion- and organ-level results are stable enough to inform course review. Supporting objectives concerned the interpretation of individual scores, the relation to a 15-item knowledge test and delivery. Results One hundred fifty-three students were scored on at least one station and 136 on both (total score 20.7 ± 4.1 of 28). The single-rater intraclass correlation coefficient (ICC) was 0.78 (95% CI 0.63–0.88; n = 39) and 0.73 (0.52–0.86; n = 33), and 0.65 and 0.69 after adjustment for organ; within-station Cronbach’s alpha was 0.70 and 0.69, and 0.34 for the four interpretive criteria. Dynamic scanning criteria were hardest and among the most discriminating, basic criteria showed ceiling effects, and the difficulty ranking was concordant across cohorts (Kendall’s W = 0.92). The allocated organ accounted for 22% of score variance, examiners for 7% in a crossed model. Between-station consistency was low ( r = 0.17; organ-adjusted ICC 0.35); a reliability of 0.70 would require 5 organ-standardised stations (range 2 to 10). The total score correlated with the post-course knowledge test ( r = 0.38; n = 79). Conclusions Aggregated criterion- and organ-level results were reproducible across cohorts and can inform iterative review of a student-led course, within limits of precision, representativeness and transferability. Individual total scores from two stations are not reliable enough for consequential decisions about learners, regardless of agreement between examiners. Readiness for independent clinical scanning is not addressed.

BMC Medical EducationVol. 26(1)
University of Bonn (DE), Witten/Herdecke University (DE), University of Cologne (DE), University Hospital Bonn (DE), Marien Hospital Witten (DE), University Hospital Cologne (DE)
Quality education, Good health and well-being
Innovations in Medical Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon