Effect of limiting checklist on the validity of objective structured clinical examination: A comparative validity study

The Objective Structured Clinical Examination (OSCE) is a cornerstone of medical education that uses a structured approach to assess clinical skills and competency. A well-designed checklist is essential to enhance the validity of OSCE exams. This study aimed to determine whether a clinically discriminatory checklist (CDC) improves the validity of the OSCE compared with an assessment using the thoroughness checklist (TC), with a particular focus on clinical reasoning. Fourteen OSCE case scenarios with both TC and CDC were developed. Each case was administered to 350–1170 fourth-year medical students in nine medical schools within the Seoul-Gyeonggi-area (Korea) during their OSCEs in 2019 and 2020. We also conducted interstation examinations after standardized patient encounters to assess clinical reasoning ability. The validities of OSCE scores based on the TCs and CDCs were compared. The OSCE using a CDC (rather than a TC) enabled better item discrimination but provided a lower internal consistency coefficient and worse standard measurement error. Clinical reasoning scores derived using patient notes were significantly correlated with OSCE scores but varied according to the characteristics of each case, indicating that OSCE scores derived using CDCs did not assess clinical reasoning ability more accurately than OSCE scores obtained using TCs. This study found that using a CDC to limit checklist items did not improve OSCE validity and did not reflect clinical reasoning ability. Further development of robust assessment strategies that support and evaluate clinical reasoning abilities is needed.