From paper to pixels: a pilot evaluation of the validity and reliability of virtual objective structured clinical examination in emergency medicine training

Abstract Background While Objective Structured Clinical Examinations (OSCE) have been used extensively in medical education, a virtual OSCE has yet to be implemented. This study aimed to design and evaluate a virtual OSCE (v-OSCE) within emergency medicine (EM). Methods This cross-sectional pilot validation study included all 12 clinical medical students participating in the emergency medicine clinical rotation from October to November 2022. Participants were recruited using a census sampling method. Assessments were conducted via the Adobe Connect virtual platform and at the virtual examination center of Jahrom University of Medical Sciences, using 12 stations/rooms equipped with the required hardware and software facilities. Content validity, criterion validity, and reliability were evaluated, and the correlation between the scores of each station and the total score was calculated. Data were analyzed using SPSS version 24. Results The virtual OSCE consisted of 12 stations, including one rest station and 11 selected topics in EM. The correlation between students’ scores at the end of their ward rotation and their scores on the v-OSCE test was positive and significant ( r = 0.644, P = 0.044), confirming the criterion validity of the test. Additionally, there was a positive and significant correlation between even and odd station scores ( r = 0.71, P = 0.02). The correlation among assessors’ scores was also positive and significant for stations involving disease ( P ≥ 0.03). Conclusion The results indicate that v-OSCE is a viable assessment tool in emergency medicine, demonstrating acceptable validity and reliability. Therefore, this method can be considered an effective approach for evaluating students.