A Three-Year Generalizability Study of Clinical Skills Assessment: Improvement in Reliability Following Expansion of Clinical Reasoning Tasks
Abstract Background As competency-based education has expanded among health professionals, greater emphasis has been placed on teaching and assessing clinical skills as a domain distinct from knowledge. Given that clinical skills assessments involve multiple facets, establishing their validity and reliability is crucial. This study evaluated the validity and reliability of a summative clinical skill assessment (S-CSA) at a single college over three consecutive years (2023–2025). Methods The S-CSA consists of clinical performance examinations (CPX) using patient-encounter cases and objective structured clinical examinations (OSCE) using clinical skill cases. Data from three years were analyzed, with 47, 46, and 56 examinees in 2023, 2024, and 2025, respectively. The analyses included descriptive statistics, correlations among CPX domain scores, CPX total scores, and OSCE total scores, and estimation of generalizability coefficients using generalizability theory (G-theory). Results Descriptive analyses showed a consistent tendency for CPX scores to be lower than OSCE scores. In particular, the history-taking and physical examination domains tended to show lower achievement levels across the years. Correlations among CPX domains were modest to moderate, whereas correlations between OSCE and CPX total scores remained low, suggesting that the CPX and OSCE assessed distinct constructs. The G-coefficients ranged from 0.534 to 0.850. Reliability estimates varied across cohorts but became more stable over time, with fewer poorly functioning stations identified in later years. Sensitivity analyses demonstrated that the exclusion of underperforming stations substantially improved reliability estimates in earlier cohorts. Conclusions Over three years, repeated analyses of the reliability of the clinical skills assessment showed that the examination ultimately reached a level generally considered acceptable for high-stakes assessment. This improvement may have been supported by the identification of stations that undermined the overall reliability and by the increasing number of CPX stations emphasizing clinical reasoning. Continued case development and further refinement are required. Future studies should include item-level reviews and analyses with larger sample sizes.
Authors
- Eui-Hyoung Hwang (ORCID: https://orcid.org/0000-0003-0460-3194)
- Hye-Yoon Lee (ORCID: https://orcid.org/0000-0002-9486-1703)
- Minna Hong
- Soo-Hyun Lee
- Seong-Eun Kwon
- Man-Suk Hwang
- Jihwan Kim
- Aram Jeong
Institutions
- Gachon University (KR)
- Pusan National University Korean Medicine Hospital (KR)
- Pusan National University (KR)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-13
- DOI
- https://doi.org/10.21203/rs.3.rs-10539676/v1
- Primary Topic
- Innovations in Medical Education
- Type
- preprint
Funders
- National Research Foundation of Korea
- Ministry of Science and ICT, South Korea