A Three-Year Generalizability Study of Clinical Skills Assessment: Improvement in Reliability Following Expansion of Clinical Reasoning Tasks

Abstract Background As competency-based education has expanded among health professionals, greater emphasis has been placed on teaching and assessing clinical skills as a domain distinct from knowledge. Given that clinical skills assessments involve multiple facets, establishing their validity and reliability is crucial. This study evaluated the validity and reliability of a summative clinical skill assessment (S-CSA) at a single college over three consecutive years (2023–2025). Methods The S-CSA consists of clinical performance examinations (CPX) using patient-encounter cases and objective structured clinical examinations (OSCE) using clinical skill cases. Data from three years were analyzed, with 47, 46, and 56 examinees in 2023, 2024, and 2025, respectively. The analyses included descriptive statistics, correlations among CPX domain scores, CPX total scores, and OSCE total scores, and estimation of generalizability coefficients using generalizability theory (G-theory). Results Descriptive analyses showed a consistent tendency for CPX scores to be lower than OSCE scores. In particular, the history-taking and physical examination domains tended to show lower achievement levels across the years. Correlations among CPX domains were modest to moderate, whereas correlations between OSCE and CPX total scores remained low, suggesting that the CPX and OSCE assessed distinct constructs. The G-coefficients ranged from 0.534 to 0.850. Reliability estimates varied across cohorts but became more stable over time, with fewer poorly functioning stations identified in later years. Sensitivity analyses demonstrated that the exclusion of underperforming stations substantially improved reliability estimates in earlier cohorts. Conclusions Over three years, repeated analyses of the reliability of the clinical skills assessment showed that the examination ultimately reached a level generally considered acceptable for high-stakes assessment. This improvement may have been supported by the identification of stations that undermined the overall reliability and by the increasing number of CPX stations emphasizing clinical reasoning. Continued case development and further refinement are required. Future studies should include item-level reviews and analyses with larger sample sizes.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-10539676/v1
Primary Topic
Innovations in Medical Education
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

A Three-Year Generalizability Study of Clinical Skills Assessment: Improvement in Reliability Following Expansion of Clinical Reasoning Tasks

Eui-Hyoung Hwang, Hye-Yoon Lee, Minna Hong, Soo-Hyun Lee et al.
Research Square
Innovations in Medical Education
preprint

A Three-Year Generalizability Study of Clinical Skills Assessment: Improvement in Reliability Following Expansion of Clinical Reasoning Tasks

Eui-Hyoung Hwang, Hye-Yoon Lee, Minna Hong, Soo-Hyun Lee, Seong-Eun Kwon, Man-Suk Hwang, Jihwan Kim, Aram Jeong
preprint en

Abstract

Abstract Background As competency-based education has expanded among health professionals, greater emphasis has been placed on teaching and assessing clinical skills as a domain distinct from knowledge. Given that clinical skills assessments involve multiple facets, establishing their validity and reliability is crucial. This study evaluated the validity and reliability of a summative clinical skill assessment (S-CSA) at a single college over three consecutive years (2023–2025). Methods The S-CSA consists of clinical performance examinations (CPX) using patient-encounter cases and objective structured clinical examinations (OSCE) using clinical skill cases. Data from three years were analyzed, with 47, 46, and 56 examinees in 2023, 2024, and 2025, respectively. The analyses included descriptive statistics, correlations among CPX domain scores, CPX total scores, and OSCE total scores, and estimation of generalizability coefficients using generalizability theory (G-theory). Results Descriptive analyses showed a consistent tendency for CPX scores to be lower than OSCE scores. In particular, the history-taking and physical examination domains tended to show lower achievement levels across the years. Correlations among CPX domains were modest to moderate, whereas correlations between OSCE and CPX total scores remained low, suggesting that the CPX and OSCE assessed distinct constructs. The G-coefficients ranged from 0.534 to 0.850. Reliability estimates varied across cohorts but became more stable over time, with fewer poorly functioning stations identified in later years. Sensitivity analyses demonstrated that the exclusion of underperforming stations substantially improved reliability estimates in earlier cohorts. Conclusions Over three years, repeated analyses of the reliability of the clinical skills assessment showed that the examination ultimately reached a level generally considered acceptable for high-stakes assessment. This improvement may have been supported by the identification of stations that undermined the overall reliability and by the increasing number of CPX stations emphasizing clinical reasoning. Continued case development and further refinement are required. Future studies should include item-level reviews and analyses with larger sample sizes.

Research Square
Gachon University (KR), Pusan National University Korean Medicine Hospital (KR), Pusan National University (KR)
National Research Foundation of Korea, Ministry of Science and ICT, South Korea
Good health and well-being, Quality education
Innovations in Medical Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon